What is power analysis?

In more detail

Power is the other half of the error budget. Significance controls how often a test calls a win that is not there; power controls how often it misses one that is. A conventional target is 80%, meaning a real effect of the planned size is detected four times in five.

Low power does more damage than wasted weeks. When an underpowered test does cross the line, the estimate that crossed it is usually inflated, because only an unusually lucky draw could clear the bar. The win then fails to repeat.

The calculation is cheap and belongs before the test, not in the post-mortem. If the answer is that the store cannot power the test, the honest options are a larger change, a more frequent metric, a longer window within limits, or not running it.

Key points

• Run it before the test, not after.
• Inputs: baseline rate, minimum detectable effect, significance level, traffic.
• Wins from underpowered tests tend to overstate the effect.

How Liftable relates

Sources

1. Jacob Cohen, Statistical Power Analysis for the Behavioral Sciences (2nd edition, 1988). Source of the conventional 80% power target, used alongside a 5% significance level.
2. Andrew Gelman and John Carlin, “Beyond Power Calculations” (Perspectives on Psychological Science, 2014). Shows that statistically significant results from underpowered studies tend to exaggerate the size of the effect.