Honest split testing and measurement for Shopify stores

Peeking, and why most results are not real

A fixed-horizon test read every morning and stopped on the first good day inflates its false-positive rate several times over. A sequential test (Liftable uses a mixture sequential probability ratio test) is built to be read continuously: its p-value is always valid, so a cron job can read it every hour without changing what a result means. The method page has the thresholds.

Starting a test the store cannot read

Before a test starts, a power check takes the store’s last 28 days of sessions on that surface and the metric’s variance, and computes how long it would take to detect the pre-registered effect. If that is longer than the limit, the test is refused and the page says how many sessions it would need. As a rough guide, about 50,000 sessions a month is where a storefront change can be measured in a few weeks; below that, only larger effects read in that time.

What makes a dollar figure

Winning the test is not a revenue claim. A shipped change keeps a 10% holdback on the old version for 30 days, and the receipt compares real Shopify orders between the two. Only that comparison puts a number next to a dollar sign, and refunds that arrive later shrink it. No store has a verified line on its receipt yet, and the site says so wherever it mentions one.

What Liftable does about it today

What is live today. Liftable finds the opportunity and our founding-team programme implements and measures the first fix with you. Automated testing is not live on any merchant’s store yet.

• Available now the scan, the evidence, the drafts, the approval
• With the founding team implementing and measuring the first fix
• Coming automated tests on a merchant’s store

What you get

• A pre-registration stored with every test: the metric, the minimum detectable effect, the alpha and the guardrails, before any traffic is split.
• A power check that refuses an underpowered test and says what traffic it would need, instead of letting it run for a month and read nothing.
• Guardrails on revenue per session, contribution margin, checkout errors and page speed, armed from the first read; a sample-ratio check at every read.
• A receipt that counts only tests that won, measured against a holdback on real orders, restated when refunds arrive.
• The founding team running the first measured fix with you, with a published result even when the test loses.

Who it is for

A good fit

• Brands with the traffic to read a test (about 50,000 sessions a month as a rough guide) and a team that would rather hear ‘not proven’ than a flattering number.
• Data teams who want the method in writing before an install.

Not the right tool

• Stores below the traffic line that want a weekly stream of ‘winners’. The power check will refuse most of those tests, and we tell you which side of the line you are on first.
• Teams with their own hypothesis backlog and a visual editor habit: Shopify Rollouts or Shoplift run what you design.

Questions