Calculate a p-value by hand
You've seen the picture in Lesson 03. Pictures fade. What sticks is doing the arithmetic until it's boring — the same way you learned long division. An A/B test is four numbers, three formulas, one table lookup. This page gives you one worked example and then generates fresh problems forever. Do at least five.
The entire calculation
Every A/B test result you'll ever read reduces to this. Work in percentage points (so 4.20% is the number 4.20) and it stays clean:
rate = 100 × conversions ÷ visitors
// step 2 — the gap (percentage points)
gap = rateB − rateA
// step 3 — the standard error of the gap (pp)
SE = √( rateA×(100−rateA)/nA + rateB×(100−rateB)/nB )
// step 4 — the z-score: how many SEs of gap you have
z = gap ÷ SE
// step 5 — the p-value: look |z| up in the table below
p < 0.05 (|z| > 1.96) → significant
That's it. The SE is the only formula with any meat, and it's just "how much luck-wobble should a gap between these two rates have?" — the same wobble you watched in Lesson 02, now as a number you can compute. The z-score asks: is my gap big compared to the wobble? The p-value converts that to "how often luck alone beats this."
Worked example — follow along with a calculator
100 × 200 ÷ 5000 = 4.00%. B: 245 of 5,000 → 100 × 245 ÷ 5000 = 4.90%.4.90 − 4.00 = 0.90 pp√( 4.00×96.00/5000 + 4.90×95.10/5000 ) = √( 0.0768 + 0.0932 ) = √0.1700 = 0.412 pp0.90 ÷ 0.412 = 2.180.825 (quarter the traffic → double the wobble), so z = 0.90 ÷ 0.825 = 1.09 → p ≈ 0.28. Not significant. The gap didn't change — your ability to hear it over the noise did. You just computed the thing Lesson 03 could only show you.Now you. Repeatedly.
Fresh numbers every time. Round rates and the gap to 2 decimals, SE to 3, z to 2, and read p from the table. Small rounding differences are accepted. No calculator shame — the point is the procedure, not mental arithmetic.
100 × conversions ÷ visitorsrateB − rateA (negative is allowed — B can lose)√( rA(100−rA)/n + rB(100−rB)/n )gap ÷ SEOne more formula, since you're warm: the Bayesian read
Lesson 07 will argue about philosophy; the arithmetic is almost embarrassing. With no strong prior opinion, the probability that B is genuinely better than A is the one-sided version of the same lookup:
P(B beats A) ≈ Φ(z) = the "cumulative" column a fuller z-table gives you
z = 2.18 → Φ(z) ≈ 0.985 → ~98.5% chance B really is better
z = 1.09 → Φ(z) ≈ 0.86 → 86% — promising, far from proof
Same four numbers, same z. The frequentist p asks "how often would luck fake this?"; the Bayesian number asks "given what I saw, how likely is B better?". Both are one division and one table away from your raw counts. Anyone who makes either sound like wizardry is selling something.
Honesty note: the drill uses the unpooled SE, the version Reality Check uses. Textbooks often pool the rates under the null first — the answers differ in the third decimal and never in the verdict. And the Φ(z) shortcut assumes a flat prior; a sceptical prior (the honest choice for ecommerce, see the tool) pulls that 98.5% down.
Drill until the procedure is boring, then let the machines take over: Instant Analysis runs these exact steps (plus the honesty checks) on a real CSV, and Reality Check adds the sceptical-prior version of Φ(z) you just met.