Significance and the p-value, without the fog
The p-value is the most quoted number in testing and the most misunderstood. Here it is with a picture instead of a formula. Once you can see it, you will never misread it again, and you will spot the people who do.
In Lesson 02 you watched luck open up gaps between two identical pages. Here is the useful part: those gaps are not random in a hopeless way. They have a shape. If A and B were truly the same, most tests would land near a 0% gap, a few would drift out to a small gap, and only rarely would luck produce a big one. Draw that as a curve and it looks like a bell.
That bell is the world where nothing is going on. Statisticians call it the null. Your real test gives you one number, the gap you actually measured. The whole question of significance is just: where does your gap land on that bell? Near the middle, and luck explains it easily. Way out in a thin tail, and luck almost never does that, so something real is probably happening. Play with it.
The orange tails are the p-value. Read it as one plain sentence: if A and B were really identical, luck would produce a gap this big or bigger this often. A p-value of 0.40 means "forty percent of the time, pure chance beats this," which is a shrug. A p-value of 0.02 means "only two percent of the time," which is a result worth trusting.
The famous 0.05 cutoff (the red line) is just a convention: call it "significant" when luck would beat it less than 5% of the time. That 5% is the same 1-in-20 false alarm rate you watched pile up in Lesson 02. There is nothing magic about 0.05. It is a line someone drew in 1925 and the industry never moved.
Notice the second slider. Drag visitors up and the bell gets skinnier, because more data means less luck-wobble, exactly what you saw in Lesson 01. The same +15% lift that was a shrug at 2,000 visitors slides into the tail and becomes significant at 15,000. The lift did not change. Your ability to hear it over the noise did.
What a p-value is NOT
This is where almost everyone, including people who run tests for a living, goes wrong. Three things the p-value does not tell you:
Getting these three straight is what separates reading a test from being sold one. For the full checklist to run before you trust a result someone hands you, see 7 questions to ask before you believe an A/B test.
You now know what the picture means. Next, stop looking at pictures: Lesson 04 makes you compute the p-value yourself — rates, standard error, z-score, table lookup — with fresh drills until it's boring. To pin your numbers down before a test, Lockbox does the sample size math for you. And if you are not sure your store even gets enough visitors to run a clean test in the first place, how much traffic you need to A/B test works it out with real store numbers.