Does Bayesian A/B testing fix peeking?

You question a result that landed suspiciously fast, and the answer comes back in one line: our platform is Bayesian, so checking early is fine. That answer is half right. The missing half is the half that decides whether your winner is real.

The Bayesian defence of peeking is not marketing nonsense. There is a genuine mathematical result underneath it, and the people repeating it are usually repeating something true. The trouble starts when that result gets stretched to cover a decision it was never about. Since almost nobody checks which version their tool actually implements, I ran the simulation instead of arguing about it.

01What peeking costs you

Peeking means looking at a running test and stopping it the moment the numbers look good. Every look is a fresh opportunity for random noise to wander across your threshold. Do it once and you get the false alarm rate you signed up for. Do it thirty times and you have given luck thirty chances to fool you, while still reporting the number you would have earned from a single honest look.

To measure it, I ran an A/A test: two identical versions, 2.5% conversion, 20,000 visitors per arm, so any winner is fake by construction. Then I varied how often the tester peeked, across 20,000 simulated experiments per setting.

One look gives you the 4.8% you expected. Ten looks gives you 19.4%. Fifty looks gives you 31.9%. A third of tests between two identical pages will hand you a winner if you check often enough and stop when you like what you see.

02The part the vendors get right

A frequentist p-value carries an assumption about when you were going to stop. Fix a sample size, then break that promise by stopping early, and the number stops meaning what the textbook says it means. A Bayesian posterior works differently. A prior is what you believed about the conversion rate before the data arrived, and a posterior is that belief after the data updates it. The posterior describes the evidence you are holding right now, at whatever sample size you happen to be at, with no promise about a finish line baked in. Under the likelihood principle it does not depend on why you stopped collecting.

So the vendor claim that a Bayesian number stays meaningful when you look at it early is correct. The posterior really is interpretable at any moment. That is a real property and a real advantage.

03The part they leave out

Interpretable at any moment is a weaker claim than safe to stop at any moment, and the gap between those two is where the money goes missing. A posterior that is valid at every look says nothing about what happens when your decision rule is "keep looking until one of these numbers crosses 95%, then stop." That rule is optional stopping, and optional stopping is exactly what inflates false alarms.

The purple line in the chart is a Bayesian test with the default flat prior that most tools ship with, stopped the moment the probability of being better crossed the threshold. It reaches 31.7% at fifty peeks. The frequentist rule reaches 31.9%. They are the same line, because with a flat prior the posterior is doing nearly the same arithmetic as the p-value it replaced. Swapping the label on the number changes nothing about the discipline of when you stop looking.

There is a sharper version of this. With a flat or uninformative prior and no fixed horizon, you can be guaranteed to eventually cross any threshold you pick, even when there is no effect at all. Statisticians call it sampling to a foregone conclusion. Keep peeking long enough and the tool will always find you a winner.

04What actually protects you

The green line is also Bayesian, with the same threshold and the same peeking. It barely moves: 1.6% at one look, 4.1% after fifty. The only change is the prior, which now carries real information about where the conversion rate sits rather than treating every rate from 0% to 100% as equally plausible. A prior that knows something pulls wild early swings back toward reality, so noise struggles to reach the threshold at all.

That protection is bought, not free. Notice the green line starts at 1.6% rather than 4.8%, which means it is a more conservative test that will also be slower to confirm real wins. That is the trade, and it is a reasonable one to make deliberately.

So the frequentist and Bayesian labels are not what separates the safe line from the dangerous ones. What your significance number means matters less here than what your tool does about continuous monitoring. What genuinely helps: a prior that carries information, a sequential method built for repeated looks, or a stopping point written down before the test starts.

05Before the next test

All of this is decided before the test runs, which is the inconvenient part. Lockbox pins your metric, sample size, and stopping rule up front, so peeking has nowhere to go regardless of which statistical engine sits underneath. If you are holding a marginal winner right now that was called early, Reality Check will deflate it to an honest estimate before you present it. And if you want to know whether your platform behaves the way it claims, the A/A validator runs the experiment I ran above against your own tool.

The question worth carrying into the next readout is short. When did everyone agree to stop looking, and was that written down before the test started?