Glossary · Testing · also anytime-valid p-value
What is an always-valid p-value?
An always-valid p-value is a p-value that stays valid however often it is checked and whenever the test is stopped: if there is no real effect, the chance it ever drops below the significance level is held at that level. Sequential methods such as the mSPRT produce one, which lets a running test be monitored without the false positives that peeking causes.
- Reviewed 6 October 2026
In more detail
An ordinary p-value is valid at one moment: the planned end of a fixed-horizon test. Read it every day and act on the first low value, and the real false-positive rate climbs well above the stated one.
An always-valid p-value only moves downwards as evidence accumulates, and it is calibrated against every possible stopping time at once. The price is a wider margin: at any single look it needs more evidence than a fixed-horizon p-value would.
It comes paired with always-valid confidence intervals, which narrow as data arrive and can be read at any time with the same guarantee.
Key points
- Valid at every look, not only at a planned end.
- Never rises as data accumulate.
- Needs more evidence than a fixed-horizon p-value at any single look.
How Liftable relates
Sources
- Johari, Koomen, Pekelis and Walsh, “Always Valid Inference: Continuous Monitoring of A/B Tests” (Operations Research, 2022). Defines always-valid p-values and confidence intervals for online tests, built on the mixture sequential probability ratio test, and shows they stay valid under continuous monitoring.
Related terms
Guides that go deeper
Last reviewed against the product on 6 October 2026.
Scan your store.
See where your store is losing revenue and what to fix first. It is free, takes about a minute, and needs no install.