A/B testing basics: the statistics, without the formulas
Before any of the clever stuff, the ground floor. What an A/B test actually is, what a conversion rate really tells you, and the one fact that makes the whole thing need statistics in the first place. No formulas to memorize. Just the ideas.
What an A/B test is
You have a web page. You think a change might make more people buy. So you show half your visitors the old version (A) and half the new version (B), at random, at the same time. Then you compare how each group behaved. That is the entire idea. A is the control, B is the variant, and splitting traffic randomly is what makes it a fair fight instead of a guess.
What you compare is the conversion rate: out of the people who saw a version, what share did the thing you wanted. If 2,000 people saw version A and 50 of them bought, that is a 2.5% conversion rate.
The catch: every visitor is a coin flip
Here is the fact that everything else is built on. A single visitor either converts or does not. Buy or no buy. That is a coin flip, just a very unfair one weighted to land on "no buy" about 39 times out of 40. And here is the problem with coin flips: a handful of them almost never lands on the exact average. Flip a fair coin ten times and you will rarely get exactly five heads. You get four, or seven, and it feels normal.
So the conversion rate you measure is never the "true" rate. It is the true rate plus a dose of luck. The question the whole field of statistics answers is: how big is that dose of luck, and how do I stop it from fooling me? Play with it below. This store has a true conversion rate of exactly 2.5%, but you only get to see what a sample of visitors happened to do.
Leave the slider at 100 and hit Send 30 samples. The dots splatter across the whole track. Some samples say 0%, some say 5%, from the exact same store. Now drag the slider to 5,000 and clear and do it again. The dots collapse into a tight little cloud right on 2.5%. Nothing about the store changed. You just stopped letting luck have so much room.
That is the whole reason A/B testing is a statistics problem and not just arithmetic. If you send a new page to 200 visitors and it converts at 3.1% versus the old one's 2.5%, you have learned almost nothing, because at 200 visitors the number bounces around that much on luck alone. Two things follow from this, and they are the rest of this course.
That is the foundation, and you can feel all of it in the demo above. A number from few visitors is mostly noise. More visitors squeeze the noise out. Everything else, the p-values, the power, the winner's curse, is just careful bookkeeping on top of this one idea.