What Is A/B Test Sample Size?
A/B test sample size is the number of visitors needed in each test group to detect a specified conversion-rate improvement with a chosen level of confidence. Planning the sample before launch helps prevent inconclusive tests and false wins caused by stopping when early results look favorable.
The required sample depends on your baseline conversion rate, minimum detectable effect (MDE), significance level, and statistical power. For a fixed traffic level, asking your test to detect a smaller lift means collecting more data; increasing the MDE shortens the test but may miss smaller improvements.
A/B Test Sample Size Formulas
For a two-variant test with a binary outcome such as purchase or no purchase, the sample size per variant can be estimated using:
Sample size per variant ≈
[(zα/2 × √(2p̄(1 − p̄)) + zβ × √(p₁(1 − p₁) + p₂(1 − p₂)))²]
÷ (p₂ − p₁)²
p₂ = p₁ × (1 + MDE)
p̄ = (p₁ + p₂) ÷ 2
Test duration in days = (sample size per variant × number of variants) ÷ daily visitors
Here, p₁ is the baseline conversion rate, p₂ is the target rate, zα/2 reflects the significance level, and zβ reflects statistical power. The formula assumes an even traffic split and independent visitor outcomes; NIST’s statistical handbook explains how power and sample size affect hypothesis tests.
The calculator also estimates the potential business impact:
Additional orders = visitors exposed × expected absolute conversion-rate lift
Revenue impact = additional orders × average order value
Gross profit impact = revenue impact × gross margin
These are projections, not guaranteed results. A test’s measured lift can be smaller, larger, or zero.
How to Use This Calculator
- Baseline conversion rate — Enter the current conversion rate for the page or funnel step you plan to test.
- Minimum detectable effect (MDE) — Enter the smallest relative lift worth detecting. For example, a 15% MDE means the target conversion rate is 1.15 times the baseline.
- Significance — Choose the confidence threshold for calling a result statistically significant. Use 95% for most store experiments.
- Power — Set the probability of detecting a real effect of the specified size. The common planning choice is 80%.
- Daily visitors — Enter average daily unique visitors to the page being tested, not total store sessions.
- Variants — Include the control in the count: enter 2 for A/B or 3 for A/B/C.
- Average order value — Enter the average revenue per order to estimate revenue impact.
- Gross margin — Enter gross margin as a percentage to estimate the profit impact before operating expenses.
Review the required visitors per variant and estimated duration before launching. For a separate check of results after the experiment, use the A/B Test Significance Calculator.
Step-by-Step Example
A Shopify store tests a product-page layout. Its baseline conversion rate is 3%, and the team wants to detect a 20% relative lift. They use 95% significance, 80% power, 1,500 daily visitors, two variants, a $65 average order value, and 60% gross margin.
| Calculation | Result |
|---|---|
| Target conversion rate: 3% × (1 + 20%) | 3.6% |
| Absolute lift: 3.6% − 3.0% | 0.6 percentage points |
| Approximate sample required | 14,100 visitors per variant |
| Total test visitors: 14,100 × 2 | 28,200 |
| Duration: 28,200 ÷ 1,500 | 18.8 days |
| Projected additional orders: 14,100 × 0.006 | 84.6 |
| Projected revenue: 84.6 × $65 | $5,499 |
| Projected gross profit: $5,499 × 60% | $3,299.40 |
The sample-size estimate uses a standard two-sided, two-proportion calculation and is approximate; calculator methods and rounding can produce different results. The revenue and profit figures show the potential value of the lift across one variant’s test traffic, not a promise of future performance.
How MDE Changes the Test
MDE is a planning threshold, not a prediction of how much your conversion rate will improve. The table below shows approximate two-variant planning estimates at a 2.5% baseline, 95% significance, and 80% power. It assumes 1,000 total visitors per day.
| Relative MDE | Target conversion rate | Visitors per variant | Approximate duration |
|---|---|---|---|
| 10% | 2.75% | 64,000 | 128 days |
| 15% | 2.875% | 29,000 | 58 days |
| 20% | 3.0% | 17,000 | 34 days |
Use these figures as planning guidance rather than universal industry benchmarks; the actual requirement changes with the baseline rate, test design, and allocation. The tradeoff is direct: accepting a larger detectable lift reduces the required sample, but smaller real gains may go unnoticed. Check your current rate with the Conversion Rate Calculator, then compare it with relevant conversion benchmarks before choosing a test target.
6 Tips for Planning a Better A/B Test
- Choose one primary metric. Use purchase conversion rate or another outcome tied to the test hypothesis; checking many metrics increases the chance of finding a misleading apparent win.
- Test high-traffic pages first. If a product page receives 250 visitors per day and the projected test takes months, move the experiment to a higher-traffic collection or landing page.
- Keep the allocation stable. A 50/50 split gives both variants comparable exposure; changing the split mid-test can make interpretation harder.
- Include a full weekly cycle. Run through at least two complete business weeks so weekdays and weekends are represented, even if the sample threshold is reached sooner.
- Avoid overlapping changes. If you change price, imagery, and page layout at once, the test cannot identify which change caused the result.
- Check the traffic source mix. A paid campaign surge can change visitor intent; use Shopify reports or your analytics platform to confirm the audience remains comparable.
- Treat privacy as part of experiment design. Avoid collecting personal information that is not needed for the test, and follow applicable privacy requirements; the FTC’s privacy and security guidance outlines business responsibilities.
- Use qualitative research when traffic is low. Short usability tests and customer interviews can reveal confusing product details without requiring a statistically powered experiment. The SBA market research guide covers ways to understand customer demand and preferences.
Frequently Asked Questions
How do you calculate A/B test sample size?
Sample size depends on the baseline conversion rate, minimum detectable effect (MDE), significance level, and statistical power. The calculator uses these inputs to estimate visitors per variant; as a rough guide, a 2.5% baseline and 10% relative MDE can require about 64,000 visitors per variant at 95% significance and 80% power under a standard two-sided calculation. The exact result depends on the test method and assumptions.
What is minimum detectable effect (MDE)?
MDE is the smallest improvement your test is designed to reliably detect. A 10% relative MDE on a 2.5% baseline means targeting 2.75%, an increase of 0.25 percentage points; choosing a smaller MDE requires a larger sample. For many ecommerce tests, a 10–20% relative MDE is a practical starting range, while lower targets can demand substantial traffic.
How long should I run an A/B test?
Run the test until it reaches its required sample size and has covered at least two full business weeks. Do not stop early because one variant temporarily leads: repeated checks and early stopping can increase false-positive risk. If the projected duration is too long, test a higher-traffic page or choose a larger MDE only if that smaller sensitivity still supports a useful decision.
What significance level should I use?
Use 95% significance for most store tests, which corresponds to a 5% false-positive threshold under the statistical model. Consider a stricter threshold for consequential changes such as pricing, and keep power near 80% unless you have a reason to plan differently. A lower significance threshold reduces the sample requirement but makes false wins more likely.
Do I have enough traffic to A/B test?
You have enough traffic when the required sample can be reached in a timeframe that still makes the result useful, usually a few weeks rather than several months. Lower-traffic stores can test larger effects or prioritize pages with more visitors; when a reliable test is impractical, use interviews, usability studies, or heatmaps to identify friction. Use the Bounce Rate Impact Calculator to assess whether improving a high-exit page may be a better first step.
About This Calculator
Luis Dev Studio built this calculator for Shopify merchants to estimate A/B test sample size, duration, and potential revenue and gross profit impact. Explore Luis Dev Studio services or contact us for help planning your store experiments.
Updated by Luis Gonzalez · Mar 2, 2026