How to Calculate A/B Test Sample Size
A/B test sample size planning is mostly about deciding what size of change is worth detecting and how much certainty you want around the test. This page walks through the practical logic behind that calculation.
Start with the effect that matters
A/B test planning begins with the minimum detectable effect, not with the sample size itself. You first decide what change would actually be meaningful enough to influence a product, growth, or design decision.
Tiny effects require much larger samples to detect.
Add confidence and power
Confidence level controls how strict you want to be about random variation. Power controls how likely you are to detect a real effect if it exists. Together, they set the sensitivity of the test.
Higher standards on either measure usually increase the traffic needed.
Use a realistic baseline
Baseline conversion rate anchors the calculation. A page that already converts at 2% behaves differently from a page converting at 20%, even when the target lift looks similar in absolute points.
That is why A/B test sample size planning works best when it uses recent baseline data rather than rough guesses.
A practical planning sequence
A useful order is to define the decision threshold first, estimate a realistic baseline, choose the minimum detectable effect, and only then look at the sample size. That keeps the experiment grounded in business relevance instead of starting with traffic alone.
It also helps teams avoid designing tests that are technically valid but operationally unrealistic. If runtime will be too long, the earlier assumptions usually need revision before launch.
- Choose an effect size that would actually change a decision
- Use recent baseline data from the same funnel step
- Check runtime before committing to the experiment
- Revise the plan instead of launching an obviously underpowered test