Question 1

How do you calculate A/B test sample size?

Accepted Answer

Sample size depends on four inputs: baseline conversion rate, minimum detectable effect (MDE), significance level (typically 95%), and statistical power (typically 80%). The formula uses z-scores for significance and power with the difference between baseline and expected conversion rates. For example, with 2.5% baseline CVR and 10% MDE, you need about 30,000 visitors per variant.

Question 2

What is minimum detectable effect (MDE)?

Accepted Answer

MDE is the smallest improvement you want your test to reliably detect. A 10% MDE on a 2.5% baseline means you can detect a lift to 2.75% (0.25% absolute). Smaller MDE = larger sample = longer test. For most ecommerce tests, 10-20% relative MDE is practical. Below 5% MDE requires very high traffic and isn't worth testing for most stores.

Question 3

How long should I run an A/B test?

Accepted Answer

Run until you reach the required sample size AND at least 2 full business weeks (to account for day-of-week effects). Never stop a test early because it looks like a winner — that leads to false positives. With 1,000 daily visitors and 10% MDE, expect 2-4 weeks. With 500 daily visitors, expect 4-8 weeks. If it takes over 8 weeks, increase your MDE or test higher-traffic pages.

Question 4

What significance level should I use?

Accepted Answer

Use 95% significance for most tests — this means a 5% chance of a false positive (declaring a winner when there isn't one). For high-risk changes (redesigns, pricing), use 99%. For low-risk tests (button color, copy tweaks), 90% is acceptable. Never go below 90%. Statistical power of 80% is standard — meaning 80% chance of detecting a real effect.

Question 5

Do I have enough traffic to A/B test?

Accepted Answer

You need enough traffic to reach statistical significance in a reasonable time (2-8 weeks). With 500+ daily visitors and 15-20% MDE, most tests finish in 3-4 weeks. With 200 daily visitors, you'll need 20%+ MDE or 6-8 weeks. Below 100 daily visitors, A/B testing is impractical — use qualitative research (user testing, surveys, heatmaps) instead.

MDE (relative)	Absolute Lift	Per Variant	Total	Days
5%	+0.13%	250,920	501,840	502 days
10%Selected	+0.25%	64,218	128,436	129 days
15%	+0.38%	29,202	58,404	59 days
20%	+0.50%	16,797	33,594	34 days
25%	+0.63%	10,987	21,974	22 days

Free A/B Test Sample Size Calculator

Related Tools

Bounce Rate Impact Calculator

A/B Test Significance Calculator

CPM Calculator

CPC Calculator