Glossary · Testing · also SRM
What is sample ratio mismatch (SRM)?
Sample ratio mismatch (SRM) is a statistically significant difference between the traffic split a test intended and the split it actually received, such as an even split that arrives visibly uneven across a large sample. It points to a flaw in assignment, delivery or logging, and it invalidates the result however good the result looks.
- Reviewed 6 October 2026
In more detail
SRM is a reliable single check of whether a test can be trusted, and it is cheap to run on every read. Common causes are redirects that drop a variant, bots landing disproportionately in one arm, and caching that serves one variant more often.
The correct response is not to adjust the numbers but to discard the test and fix the cause. A mismatch means the two groups were not comparable, so no amount of analysis rescues the comparison.
Key points
- A significant deviation from the intended split.
- Causes: redirects, caching, bot skew, lost logging.
- The result is void: discard and fix, never adjust.
How Liftable relates
Sources
- Fabijan et al., “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments” (KDD, 2019). A taxonomy of SRM causes, including assignment, redirects, logging and bot traffic, from experimentation teams at Microsoft and elsewhere.
- Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments (Cambridge University Press, 2020). Standard reference on online testing practice, including guardrail metrics, sample ratio mismatch and the cost of peeking.
Related terms
Guides that go deeper
Last reviewed against the product on 6 October 2026.
Scan your store.
See where your store is losing revenue and what to fix first. It is free, takes about a minute, and needs no install.