Latest
New
Does Bayesian A/B testing fix peeking?
"Our platform is Bayesian, so checking early is fine." Half right. In a simulation of 20,000 tests between two identical pages, a default flat prior called a fake winner 31.7% of the time. The prior does the protecting, not the label.
Read the guide →7 questions to ask before you believe an A/B test
The deck says +23% and you didn't run the test. Honest tests survive seven short questions, and the others start wobbling on the second one.
How much traffic do you need to A/B test an online store?
At 10,000 visitors a month, detecting a 10% lift takes 56 weeks. The arithmetic to run before any test, and what actually works at small-store traffic.
What this blog covers
Sample size & power
Will this test ever conclude?
How to size a test before you run it, why underpowered tests waste months, and the traffic math nobody prints on the box.
→ Try Lockbox
Winner's curse
Why winners shrink after launch
Significant tests systematically overstate their effect. How to deflate a result to its honest size before you announce it.
→ Try Reality Check
Platform trust & SRM
Is your platform telling the truth?
Sample ratio mismatch, tracking loss, and the A/A test that catches a broken tool before it biases every result you read.
→ Try the Platform Validator
Program measurement
Did the wins reach revenue?
Why announced lift and realized lift diverge, and how to reconcile a year of "winning" tests against your actual numbers.
→ Try the Program Ledger
Take the checklist with you.
The 7 questions as a one-page PDF: print it, or keep it open during the next test readout. New guides land here most weeks; posts go up on LinkedIn first.