Jul 11, 2026test validity5 min read

7 questions to ask before you believe an A/B test

The deck says +23%, "98% confident", ship it. You didn't run the test and you can't re-derive the math in the meeting. You don't have to. A test that was run honestly survives seven short questions, and a test that wasn't starts wobbling on the second one. Here they are, with what good and bad answers sound like.

by Sebastian Zijlstra · more essays

Checklist graphic of seven questions to ask before believing an A/B test result: metric chosen in advance, even traffic split, sample size set up front, fixed stopping date, lift after deflation, number of metrics examined, and whether past wins reached revenue. — The seven questions. Two or three of them, asked calmly, are usually enough.

None of this requires statistics. Every question below is about process, and process is exactly what a presenter controls. The pattern to watch for is simple: honest testers answer instantly, because they decided these things before the test started. Evasive answers mean the decisions were made after the data came in, and that is where fake winners come from.

01Was this metric chosen before the test started?

A test is only valid for the question it was designed to answer. If the plan said "conversion rate" and the slide celebrates "revenue per visitor", someone went shopping after the results arrived. Measure enough numbers and one of them will clear the bar by luck alone. If the headline metric was picked after the data came in, the significance on the slide means nothing.

GOOD"Yes, it's in the test plan. Here's the doc from before launch."

BAD"We tracked several metrics and this one showed the clearest signal."

A written plan from before launch is called pre-registration. Lockbox exists so a team can produce one in five minutes.

02What were the raw visitor counts per arm?

The platform promised a 50/50 split. Ask for the actual numbers. On tens of thousands of visitors, even 51/49 can be statistically impossible by chance, and it means the assignment itself was broken: bots counted on one side, a redirect dropping people on the other. Every number downstream inherits the break. A test with a broken split is not a weaker result. It is no result.

GOOD"24,912 vs 25,090. We ran the SRM check, it passed."

BAD"We'd have to dig that export up."

SRM stands for sample ratio mismatch. Paste the two counts into the Platform Validator and it does the check in seconds.

03How many visitors did the test need, and who decided that before launch?

Every test needs a minimum amount of traffic to detect the effect it's hunting, decided up front. A test sized by the calendar is almost always undersized, and undersized tests have a nasty property: the only way they reach significance is by getting lucky, so their winners are inflated. "Two weeks, like always" is a schedule. A schedule is not a sample size.

GOOD"We needed 18,000 per arm to detect a 10% lift. Here's the calculation."

BAD"We ran it for two sprints."

04Was the stopping date fixed, or did the test end when it looked good?

Results wobble day to day. Check the dashboard every morning and stop the moment it flashes green, and you will catch a lucky peak. Done routinely, this quietly turns a 5% false-alarm rate into roughly one fake winner in four. Ending a test on its best day manufactures winners out of noise.

GOOD"Stop rule was written down up front: 4 weeks or 18,000 per arm, whichever came second."

BAD"It hit significance on day 9, so we called it early."

05What does the lift look like after deflation?

Tests that barely clear the significance bar overstate their effect, often by double. This is the winner's curse: when a modest true effect passes a noisy test, it passed because noise flattered it. The +23% on the slide is the flattered version. The smaller the test, the more the announced number exaggerates reality.

GOOD"Observed +23%, shrunk estimate +9%. We'd plan revenue on the +9%."

BAD"The tool said 23, we report 23."

You can run this one yourself in two minutes: put the deck's four numbers into Reality Check and read the honest estimate.

06How many metrics and segments were examined?

Twenty metrics at 95% confidence means one false winner is the expected outcome, not the unlucky one. The same goes for segments. A win that only exists for "mobile users, returning, DE market" was usually found by slicing until something turned green. A win discovered in the eleventh segment is a coincidence with a slide deck.

GOOD"One primary metric, two guardrails, no segment mining. Segments are exploratory only."

BAD"The overall result was flat, but look at this segment."

07Did last year's wins show up in revenue?

The program-level question, and the one that ends careers of fake winners. Add up every lift announced over the past year and compare it with what the revenue line actually did. If the claims total +40% and revenue moved +4%, the testing program produces announcements. Wins that never reach the P&L were never wins.

GOOD"We reconcile claimed lift against actuals quarterly. Here's the ledger."

BAD"Attribution makes that impossible to measure."

The Program Ledger does this reconciliation from numbers you already have.

// how to use these Don't read all seven off a list. Pick two or three and ask them calmly. Teams that test honestly answer in one sentence, because the answers existed before the test did. And you can check privately first: paste the deck's numbers into Instant Analysis before the meeting. It runs in your browser and nobody sees you checking.

// the stack

Seven free tools for honest ecommerce experimentation: platform validation, pre-registration & sample size, survival analysis, winner deflation, integrity receipts, the program ledger, and subscription valuation. All of it runs in your browser. Explore the stack →

Sebastian Zijlstra

I build tools for ecommerce experimentation that hold up under scrutiny, and write about where A/B testing quietly goes wrong. Everything on this site runs in your browser, free. See the stack, or connect on LinkedIn.