What is Statistical Significance?
A measure of confidence that an observed difference between test variants is real rather than due to random chance, typically requiring 95% confidence for A/B test decisions.
Understanding the Details
Statistical significance determines when you can trust A/B test results. If variant B shows a 5% higher conversion rate than variant A, is that a real improvement or just noise? Statistical significance answers this by calculating the probability that the observed difference occurred by chance. The standard threshold is 95% confidence, meaning there's less than a 5% chance the result is random. Reaching significance requires adequate sample size — testing with too few visitors produces unreliable results. The two biggest errors in testing are stopping tests too early (declaring winners before significance is reached) and ignoring practical significance (a statistically significant 0.1% improvement may not be worth implementing).
How It Works in Practice
A/B test conclusion
After 10,000 visitors per variant, a 12% conversion lift reaches 98% statistical significance, giving confidence to implement the change.
Premature stopping
A test shows a 20% lift after 500 visitors but only 60% significance. Waiting for 3,000 visitors reveals the true lift is only 3%.
Sample size planning
Before launching a test, a power analysis determines that 8,000 visitors per variant are needed to detect a 10% lift with 95% significance.
Why It Matters
Without statistical rigour, A/B testing becomes guesswork. Statistical significance prevents making changes based on random noise, protecting against false improvements.
What People Often Get Wrong
Statistical significance means the result is important. Actually, a tiny insignificant improvement can be statistically significant with enough data.
You can check significance daily and stop when it's reached. Actually, repeated checking inflates false positive rates — tests need predetermined durations.
95% significance means 95% chance the winner is better. Actually, it means there's less than 5% probability the difference is due to chance.
How We Handle Statistical Significance
We design tests with proper sample size calculations and run them to significance before making decisions, ensuring changes are based on real evidence.
Common Questions
Need Help With Statistical Significance?
If you'd like to discuss how statistical significance applies to your business, we're happy to explain further.