Glossary · Testing
What is statistical significance?
Statistical significance is the judgement that an observed difference would be unlikely to arise by chance if there were no real effect. It says nothing about how large the effect is, whether it will last, or whether it is worth acting on, which are separate questions that significance is often mistaken for.
- Reviewed 6 October 2026
In more detail
The most damaging misuse in ecommerce is peeking: watching a running test and stopping it the moment it crosses a threshold. With a fixed-horizon test that inflates the false-positive rate substantially, because you have given yourself many chances to cross the line by chance.
Either fix the sample size in advance and do not look, or use a method designed to be monitored continuously. Doing neither, which is the default, produces a stream of wins that do not repeat.
Key points
- Not a measure of effect size or of business value.
- Peeking at fixed-horizon tests inflates false positives.
- Either commit to a sample size in advance or use a sequential method.
How Liftable relates
Related terms
Guides that go deeper
Last reviewed against the product on 6 October 2026.
Scan your store.
See where your store is losing revenue and what to fix first. It is free, takes about a minute, and needs no install.