Sample Size
Sample size is the number of visitors or observations included in each variant of a test, large enough to detect a meaningful difference with acceptable reliability.
In depth
Required sample size is determined before launch through a power calculation that combines your baseline conversion rate, the minimum detectable effect you care about, and your chosen confidence level and statistical power. Smaller effects, lower baselines, and higher confidence all push the required number of visitors up, sometimes dramatically, which is why ambitious tests on low-traffic pages can take months.
The most common pitfall is under-powering: ending a test with too few visitors produces noisy results that swing between runs and tempt teams to ship the wrong variant. In a quiz-funnel context, calculating sample size up front sets a clear finish line, prevents premature peeking, and tells you whether a planned test is even feasible given your weekly traffic before you invest design and engineering time.
Example in practice
Frequently asked questions
How do I calculate the right sample size?
Use a power calculator that takes your baseline conversion rate, minimum detectable effect, confidence level, and statistical power. It returns the visitors needed per variant before you launch the test.
What happens if my sample size is too small?
An under-powered test produces noisy, unstable results that can falsely declare a winner or miss a real difference. You risk shipping changes that do not actually improve conversions.
Can I run an A/B test on a low-traffic quiz funnel?
You can, but small effects may require more traffic than you generate in a reasonable timeframe. Calculate sample size first to confirm the test is feasible, or focus on bigger, bolder changes.