Significance and the p-value, without the fog

In Lesson 02 you watched luck open up gaps between two identical pages. Here is the useful part: those gaps are not random in a hopeless way. They have a shape. If A and B were truly the same, most tests would land near a 0% gap, a few would drift out to a small gap, and only rarely would luck produce a big one. Draw that as a curve and it looks like a bell.

That bell is the world where nothing is going on. Statisticians call it the null. Your real test gives you one number, the gap you actually measured. The whole question of significance is just: where does your gap land on that bell? Near the middle, and luck explains it easily. Way out in a thin tail, and luck almost never does that, so something real is probably happening. Play with it.

The orange tails are the p-value. Read it as one plain sentence: if A and B were really identical, luck would produce a gap this big or bigger this often. A p-value of 0.40 means "forty percent of the time, pure chance beats this," which is a shrug. A p-value of 0.02 means "only two percent of the time," which is a result worth trusting.

The famous 0.05 cutoff (the red line) is just a convention: call it "significant" when luck would beat it less than 5% of the time. That 5% is the same 1-in-20 false alarm rate you watched pile up in Lesson 02. There is nothing magic about 0.05. It is a line someone drew in 1925 and the industry never moved.

Notice the second slider. Drag visitors up and the bell gets skinnier, because more data means less luck-wobble, exactly what you saw in Lesson 01. The same +15% lift that was a shrug at 2,000 visitors slides into the tail and becomes significant at 15,000. The lift did not change. Your ability to hear it over the noise did.

What a p-value is NOT

This is where almost everyone, including people who run tests for a living, goes wrong. Three things the p-value does not tell you: