How much traffic do you need to A/B test an online store?

Some version of this question shows up every week on r/shopify, r/CRO, and r/ecommerce: "My store does about 10,000 visitors a month. Everyone says test everything, but my tests never reach significance. Am I doing something wrong?" No. The math was against you before you started.

Nobody selling testing software has much reason to answer this honestly. The real answer: below a certain amount of traffic, an A/B test can't do its one job, telling a real improvement apart from random luck. The good news is you can find your store's cutoff in about two minutes. Two words carry the rest.

01Run the arithmetic before the test

How long a test runs depends on four things: your baseline conversion rate (the share of visitors who buy today), the lift you want to catch (a "10% lift" means a 2.5% rate climbing to about 2.75%, not adding ten percentage points), and how careful you want to be about two mistakes, calling a dud a winner and missing a real one (the usual defaults, 95% confidence and 80% power). Plug in a typical small store, 2.5% conversion and 10,000 visitors a month split between two versions, and here is what comes out.

To reliably catch a 10% lift, a good result for a typical tweak, you need about 128,000 visitors. At 10,000 a month that is 56 weeks, over a year frozen on one test. A 20% lift still needs about 15 weeks; only past 30% does the timeline drop under eight weeks, which is roughly the longest a test should run before cookies clear and seasons shift. Flip it around and it is starker: catching a 10% winner inside a single month needs about 140,000 visitors a month. Reliably spotting small wins is a big-store luxury, and it is on the box of no testing tool.

02Running it anyway backfires

At low traffic the temptation is to run it and hope. But when a weak test does scrape past "significant," it is usually luck that pushed it over, so the win it reports comes out inflated, often two or three times. That is the winner's curse: the +14% you would announce is probably a +5% in a costume, and shipping it turns a coin flip into false confidence that gets budget. If you have a borderline winner in front of you now, Reality Check shrinks it to an honest number before you present it.

03What actually works at low traffic

The line in that chart is not fixed. None of these need more visitors:

• Test bigger swings. Changes that can plausibly move conversion 30% or more: the offer, how pricing is shown, bundling, a rebuilt product page. Button-color tests are for stores with a million visitors.
• Measure something more common. Far more people add to cart than check out, so counting add-to-carts gathers data much faster, often cutting the timeline by two thirds. Just commit to it up front, since it is a stand-in for the real goal.
• Pool similar pages. One test across all your product pages reaches a verdict where fifty separate per-page tests never would.
• Two versions, never five. Every extra variant splits your traffic thinner and raises the odds one looks like a winner by chance.

04Two minutes before you start

All of this only pays off if it is decided before the test runs. Lockbox sizes the test and locks your metric and stopping rule up front, so there is nothing to fudge later. Ship a winner, then log it in the Program Ledger against your real monthly numbers to see whether the wins reach revenue. Both are free and run in your browser. Two minutes of math beats three months of arguing.