Why one A/B test can lie to you

This is the single most important idea in A/B testing, and it takes thirty seconds to feel instead of thirty pages to read. Every visitor is a coin flip: they buy or they don't. Even when two pages convert at the exact same true rate, the handful who happen to buy will not split perfectly evenly. One side gets a few lucky sales, its measured rate ticks up, and suddenly it looks like a winner. Nothing changed. That is just noise.

Hit Run one A/A test a few times. Both arms are set to the same true conversion rate, so any gap you see is pure chance.

Keep running and the false-winner rate settles near 5%. That is not a bug. It is the deal you signed. "Significant at 95% confidence" literally means we accept being fooled by noise about 1 time in 20. Run twenty A/A tests and, on average, one of them hands you a confident, statistically significant, completely fake winner.

Now crank the visitors slider up. The individual gaps get smaller, but that 5% false-winner rate barely moves. More traffic buys you precision, not immunity. A big test lies less wildly, but it still lies 1 time in 20 at a single look.

Now watch peeking make it much worse

Nobody runs a test and looks exactly once. They check the dashboard every day and stop when it looks good. That feels responsible. It is the fastest way to fool yourself. Below, the simulator checks the same identical-arms test once a "day" for two weeks and stops the moment it sees significance, exactly like a person watching a live dashboard.

Why this is the whole game

Every tool on this site exists to survive the thing you just watched. If a single test can invent a winner out of noise, then a screenshot of a green dashboard proves nothing on its own. You need to know the test was big enough to hear a real signal, that nobody peeked, and that the lift was not just a lucky draw. That is the difference between a number and a number you can defend in a room.

From here the path gets concrete: how big the signal needs to be before it is worth chasing, how much traffic that takes, and how to shrink a lucky winner back to its honest size. But it all rests on this one lesson. A result is not true because it is significant. It is true because it would survive being tested again.