How to split-test when you don’t have the traffic

Why purchase-level tests fail on smaller stores

Statistical power depends on the base rate and the number of events. A purchase is a rare event, so telling a realistic effect apart from noise takes more orders per variant than many stores collect in a month.

The failure is not usually visible as a failure. The test runs, the numbers wobble, someone declares a winner, and the effect never shows up in revenue. That is the low-traffic failure mode: not inconclusive results, but confident wrong ones.

The judging-step trade, stated honestly

Judging at add to cart instead of purchase measures a narrower question: did the change make people more likely to add to cart, not did it make the store more money. Those usually move together and occasionally do not.

That is what the holdback is for. The step-level test decides relatively quickly whether the change did anything; the holdback decides slowly whether it was worth money. Neither alone is sufficient, and together they are honest.

Things that quietly invalidate a test

Peeking at a fixed-horizon test and stopping on a threshold crossing: the most common cause of wins that do not replicate.

Sample ratio mismatch, where the realised split differs from the intended one, usually because of redirects, caching or bot traffic. It means the groups were not comparable and no analysis rescues it.

Running two tests on the same surface at once, which makes both results unattributable.

When not to test at all

A control that does nothing when tapped does not need a test. Neither does a broken discount field or a script from an app you uninstalled last year.

Testing is for changes where reasonable people disagree about the outcome. Fixing something broken is not one of those, and testing it only delays the fix.

How Liftable handles a store that cannot read a test

Liftable runs a power check before any test starts, on your store’s own recent sessions, and refuses a test your traffic cannot read within its 21-day maximum. Where it can, a refusal comes with a readable alternative: the same test measured on reaching checkout or on add to cart, with how long that would take. It never switches the metric on its own; you choose.

Tests are read with a sequential method (mSPRT) set to a 5% false-positive rate, checked for sample ratio mismatch on every read, and a shipped change keeps 10% of visitors on the original so revenue can be measured on Shopify orders against them. This is built, and the statistics are checked in simulation. No test has been read to a result on a merchant’s store.

As a rough guide, about 50,000 sessions a month is where a storefront change can be measured in a few weeks. Below that, only larger effects read in that time, and Liftable tells you which side of the line your store is on before you commit to anything.

Testing at the traffic you have

1. 01

   Work out the detectable effect before you start

   Compute the smallest lift your traffic could tell apart from noise at your baseline rate and planned duration. If it comes back far larger than any change you could plausibly make, the test cannot conclude, and you have saved yourself weeks.

2. 02

   Move the judging step to where the change acts

   Judge a product-page change at add to cart and a cart change at reaching checkout. The event rate rises sharply and the detectable effect shrinks with it.

3. 03

   Use a sequential method so watching is allowed

   A fixed-horizon test is invalidated by checking it early and stopping on a good day. A sequential method stays valid under continuous monitoring, so you can stop as soon as the evidence is sufficient rather than waiting out a pre-computed sample.

4. 04

   Check for sample ratio mismatch before believing anything

   If the realised split differs significantly from the intended one, the groups were not comparable and the result is void. Discard it and fix the cause; never adjust the numbers.

5. 05

   Ship behind a holdback

   Keep a small share of visitors on the original after the winner goes live, and compare afterwards. This is what catches novelty effects and decay, and it is what makes a revenue claim defensible.

Sources

1. Liftable: what is built today. Liftable’s own account of what the product does, checked against the code on 2 October 2026 and shown on /how-it-works and /compare. Every figure this page gives about Liftable comes from it. No test has run on a merchant’s store and there are no published merchant results yet.

Questions