What is the mixture sequential probability ratio test (mSPRT)?

In more detail

Abraham Wald’s sequential probability ratio test, published in 1945, compares two exact hypotheses: no effect, or one specific effect. That is efficient when the alternative is known, and a poor fit for product tests, where nobody knows the effect size in advance.

The mixture version averages the likelihood over a distribution of effect sizes, the mixing distribution, rather than betting on one. The resulting ratio can be checked after every batch, and stopping when it exceeds one over the significance level keeps the false-positive rate at that level.

In practice the mixing distribution is tuned to the effects a team expects to see. A wide one is robust; a narrow one centred on realistic effects reaches a decision sooner when it is right.

Key points

• Checks the evidence after every batch, with no penalty for looking.
• Averages over plausible effect sizes instead of assuming one.
• Produces always-valid p-values and confidence intervals.

How Liftable relates

Sources

1. Abraham Wald, “Sequential Tests of Statistical Hypotheses” (Annals of Mathematical Statistics, 1945). The original sequential probability ratio test.
2. Herbert Robbins, “Statistical Methods Related to the Law of the Iterated Logarithm” (Annals of Mathematical Statistics, 1970). Introduces mixture likelihood ratios, the basis of the mSPRT.
3. Johari, Koomen, Pekelis and Walsh, “Always Valid Inference: Continuous Monitoring of A/B Tests” (Operations Research, 2022). Defines always-valid p-values and confidence intervals for online tests, built on the mixture sequential probability ratio test, and shows they stay valid under continuous monitoring.