Confidence Level
A confidence level is the percentage expressing how often a test's confidence interval would contain the true value if the experiment were repeated, commonly set at 95%.
In depth
Confidence level is the complement of your significance threshold: a 95% confidence level corresponds to an alpha of 0.05. It frames results as an interval rather than a single number, telling you the plausible range for the true conversion difference and how reliably that range captures reality across repeated tests.
A frequent misunderstanding is reading 95% confidence as a 95% probability that variant B is better; it actually describes the long-run behavior of the method, not a single test outcome. Choosing the level is a trade-off: higher confidence reduces false positives but demands more traffic and time. For a quiz funnel with modest weekly volume, insisting on 99% confidence can stall optimization, so teams balance rigor against the cost of slower iteration.
Example in practice
Frequently asked questions
What is the difference between confidence level and statistical significance?
They are two sides of the same coin: a 95% confidence level corresponds to a 0.05 significance threshold. Confidence level frames reliability as a percentage and an interval, while significance frames it as a p-value cutoff.
Does 95% confidence mean variant B has a 95% chance of being better?
No, that is a common misinterpretation. It describes how often the method's interval would capture the true value over many repeats, not the probability for any single test.
Should I always use the highest possible confidence level?
Not necessarily, because higher confidence requires much more traffic and slows your testing cadence. Reserve 99% for high-risk decisions and use 95% for routine quiz-funnel experiments.