The peeking problem

Overview

Lessons 2 and 3 gave you a complete discipline: define the MDE, calculate sampleSize(), and commit to a concrete number of users per variant before running the experiment. This lesson presents the most common pitfall that breaks that discipline, almost always with nobody realizing they're breaking it: peeking — looking at the experiment's result before reaching the planned sample size, and stopping it the moment the p-value crosses the 0.05 threshold.

It sounds harmless. A product manager opens the experiment's dashboard on Tuesday of the second week, sees variant already ahead with a p-value of 0.04, and thinks: "we already have significance, why wait the full 6 weeks?". The problem isn't the curiosity of looking —checking a dashboard doesn't break anything by itself—. The problem is stopping there, using that first moment of significance as the signal to end the experiment. Every time you look at the result midway with the option to stop if you like what you see, you're giving random noise a new chance to cross the 0.05 threshold, purely by coincidence — and the more chances you give it, the more likely noise is to win, even with no real effect at all.

How this connects to the module. sampleSize()'s alpha=0.05 (lesson 3) and abTest()'s z-test (module 6) is a very specific promise: if you run the experiment once, up to the planned n, and look at the result once at the end, the false-positive risk is 5%. Peeking breaks exactly that condition —"once, at the end"— and this lesson builds a model that shows, with numbers, how much that 5% inflates when you break it.

An analogy: declaring a winner by checking the scoreboard every minute

Picture a close basketball game, where both teams keep trading the lead the whole time. If you check the scoreboard every minute and, the moment your team is ahead by any margin, you run off declaring "we won!" and turn off the TV — you're going to "win" many times during the game without that meaning anything at all about who actually wins. Close games' scores go up and down constantly from pure game variation; almost any team is going to be ahead at some point, even if it ends up losing. The only reliable way to know who won is to wait for the final whistle and check the score once, at that moment — not average, or count, how many individual minutes each team was ahead during the game.

An A/B test that keeps running while you check it often behaves exactly like a close game's scoreboard: the p-value goes up and down naturally, crossing the 0.05 threshold downward and upward several times, even when there's no real effect involved — pure statistical variation, the same kind of noise that makes a team be ahead "by coincidence" for a couple of minutes. Stopping the experiment the moment you see the p-value cross 0.05 for the first time is exactly the same as declaring a winner the instant your team takes the lead for a moment: you're treating a moment of normal noise as if it were the final result.

Worked example: how much the false positive grows with each look

Let's build a simplified model —a pedagogical approximation, not the full rigorous treatment— of how the accumulated false-positive risk grows when you look at the experiment several times and stop at the first one that crosses alpha=0.05. Assuming (in simplified terms) each look is roughly independent of the previous ones, the probability that at least one of k looks crosses the threshold by pure chance, even with no real effect at all, is:

P(at least 1 false positive in k looks) = 1 - (1 - alpha)^k

// peekingFalsePositiveRate: a PEDAGOGICAL approximation of how the accumulated
// false positive rate grows when looking at an experiment k times and stopping
// at the first look with p < alpha. Assumes rough independence between looks --
// a real simplification; the peeking problem in practice (continuous monitoring)
// is subtler and, in general, WORSE than this bound. See Evan Miller in the
// Resources for the full rigorous treatment.
function peekingFalsePositiveRate(alpha, k) {
  return 1 - Math.pow(1 - alpha, k);
}

console.log('=== Peeking: accumulated false positive, alpha=0.05 per look ===\n');
[1, 2, 3, 5, 10, 20].forEach((k) => {
  const rate = peekingFalsePositiveRate(0.05, k);
  console.log('k=' + String(k).padStart(2) + ' looks  ->  P(at least 1 false positive) = ' +
    (rate * 100).toFixed(2) + '%');
});

What to expect. When you run the file with Node, the output is exactly this:

=== Peeking: accumulated false positive, alpha=0.05 per look ===

k= 1 looks  ->  P(at least 1 false positive) = 5.00%
k= 2 looks  ->  P(at least 1 false positive) = 9.75%
k= 3 looks  ->  P(at least 1 false positive) = 14.26%
k= 5 looks  ->  P(at least 1 false positive) = 22.62%
k=10 looks  ->  P(at least 1 false positive) = 40.13%
k=20 looks  ->  P(at least 1 false positive) = 64.15%

With a single look at the end —lessons 2 and 3's discipline—, the false-positive risk is exactly the 5% alpha promises. But if the same team checks the dashboard every week during a 5-week experiment (k=5), that risk almost quadruples, to 22.62%. And if someone checks the result every workday for two weeks (k=10), the risk of ending up "declaring a winner" from pure noise reaches 40% — more than eight times the 5% the team believes it's accepting. The team keeps believing it's working with alpha=0.05, with all the confidence that number conveys, while the real risk it's running is much higher — with nobody having made that decision on purpose.

Why the real problem is even worse than this approximation

Today's model assumes each look is roughly independent of the previous ones — a useful simplification for seeing the general pattern, but not the full reality. In practice, consecutive looks at the same experiment are correlated (this week's p-value depends on the same accumulated data as last week's, plus the new data), and the real mathematical phenomenon —known as repeated significance testing— is more severe, not less, than this simplified bound. Evan Miller, in this lesson's reference article, shows with simulations that continuously checking an experiment (not just at a handful of discrete moments, but all the time) can push the real false-positive rate far above the nominal 5% even with relatively few effective "looks" — and, in the limit, if you monitor an experiment with no real effect for long enough, the probability it crosses the significance threshold at some point approaches near certainty. Today's 1 - (1-alpha)^k bound is useful for intuition —the more times you look and can stop, the more the risk grows— but don't treat it as the exact number for any real experiment: for that, the correct solution is a sequential-testing design (out of this guide's scope, cited in the Resources), or, more simply, lessons 2 and 3's discipline: decide the n before starting, and look at the result only once, when you reach it.

Common mistakes

Checking the experiment's dashboard often, and stopping it the moment a p-value under 0.05 shows up. What happens: someone on the team checks recommendations's result every few days during the 6-week window, and in week 3 sees pValue=0.03 —still not having reached the n calculated in lesson 3— and decides to end the experiment there, "since it already came out significant". Why it happens: waiting until the end feels slow and inefficient when the result "already looks good", and nothing on the dashboard warns that looking too early changes the real error risk. How to spot it: the experiment ended before reaching the n per variant sampleSize() had calculated, and the reason was "we already saw significance", not "we reached the planned sample". How to fix it: as today's table shows, each additional look with the option to stop inflates the real risk far above the declared 5%. The simple rule: define n with sampleSize() (lesson 3) and don't make any decision to stop the experiment until you reach it — checking the dashboard out of curiosity is fine; acting on what you see before that, isn't.

Confusing "extending the duration because it didn't reach the planned n" with "keep looking until it's convenient to stop". What happens: a team, seeing the experiment doesn't reach significance by the expected date, decides to let it run one more week —which, done with the right intent, is perfectly valid— but does it repeatedly, checking every few days whether "it's there yet" and extending only when the current result doesn't suit them. Why it happens: the difference between "keep running until the planned n" (correct) and "keep looking and stop as soon as the result looks good" (peeking) is subtle, and both feel, from the outside, like "waiting a bit longer". How to spot it: the criterion for when to stop the experiment changes based on what the dashboard shows at each check, instead of being fixed in advance at a concrete n or date. How to fix it: fix the n (or the equivalent date) before starting, and the only valid reason to end the experiment is having reached that number — never "because today's result looks good" nor "because today's result looks bad".

Believing that declaring alpha=0.05 in the code automatically guarantees a real 5% risk. What happens: a report cites alpha=0.05 as if it were a fixed, guaranteed property of the experiment, with no mention of how many times the result was checked over the window. Why it happens: alpha=0.05 shows up as an explicit parameter in sampleSize() and in abTest(), and it feels like an automatic mathematical guarantee, independent of how the experiment was actually used in practice. How to spot it: the report never mentions the experiment's monitoring process —how many times it was checked, when it was decided to end it—. How to fix it: as today's table shows, alpha=0.05 is only a real guarantee if the experiment was checked once, upon reaching the planned n. Any result report should be able to answer, with the same clarity it reports the p-value: "how many times was this experiment checked before deciding to end it?".

Exercises

Exercise 1 — Calculate the risk of a real scenario. An experiment runs for 8 weeks, and someone on the team checks the dashboard once a week (8 looks total), with the option to stop it at any of those checks if they see pValue < 0.05. Using today's approximation, what's the accumulated false-positive risk?

See solution

peekingFalsePositiveRate(0.05, 8) = 1 - (1 - 0.05)^8 = 1 - 0.95^8 ≈ 1 - 0.6634 ≈ 0.3366, that is, roughly 33.66% — more than six times the nominal 5% the team believes it's accepting. Running peekingFalsePositiveRate(0.05, 8) in Node confirms this number.

Exercise 2 — Tell peeking apart from a valid extension. The recommendations experiment doesn't reach the n=12,997 calculated for MDE=20% by the end of the planned week 6. The team decides, before looking at that week's result, to extend the experiment exactly 1 more week, committing to analyze the result only upon reaching the target n, regardless of what the dashboard shows in the meantime. Is this peeking? Justify your answer.

See solution

This isn't peeking. The decision to extend was made for an objective reason fixed in advance —the planned n wasn't reached— not from having looked at the p-value and deciding to continue only because the current result wasn't convenient. The key rule, as in this lesson's second common mistake, is that the criterion for when to stop the experiment —reaching n=12,997— never changed; what changed was how long it took to get there, a decision made without looking at the intermediate result. Peeking would instead be checking the dashboard every week during that extension and stopping the experiment the moment the p-value crossed 0.05, before reaching the target n.

Exercise 3 — Propose a correct alternative for monitoring without falling into peeking. The Mercado team wants to be able to check the experiment's progress every week —for example, to confirm the assignment stays balanced, or that there's no technical instrumentation error— without falling into this lesson's peeking risk. What would you recommend?

See solution

It's perfectly valid to check the dashboard every week to verify operational health —assignment balance, traffic volume, instrumentation errors— as long as that check doesn't include a decision to stop the experiment based on the p-value. The concrete recommendation: explicitly separate "health monitoring" (is the experiment working correctly?) from "result analysis" (did variant win?), and commit the team, in writing, to result analysis happening only once, upon reaching the n planned in lesson 3 — never before, regardless of what the health checks show along the way. If the team genuinely needs to be able to decide before the planned n, the statistically correct alternative is designing the experiment from the start as a sequential test (cited in today's Resources), not simply checking a fixed-size test often and stopping whenever convenient.

Summary and next step

In this lesson you saw the first —and probably most common— of the pitfalls that can invalidate a result, even with the correctly calculated n: peeking. You built peekingFalsePositiveRate(), a simplified pedagogical model showing how the real false-positive risk grows with each additional look —from 5% with a single look at the end, to 22.62% with 5 looks, to 40.13% with 10— and saw why the real phenomenon (continuous monitoring) is even more severe than this discrete approximation.

Before moving on you should be able to: explain why checking an experiment several times with the option to stop it inflates the false positive above the declared alpha; tell a valid extension of an experiment's duration (decided without looking at the result) apart from real peeking; and calculate the approximate accumulated risk given a number of looks.

Lesson 5 presents a pitfall with the same underlying mathematical structure, but applied to a different scenario: instead of looking at the same test several times over time, what happens when you test several different metrics at once, over the same data?

Resources

• Evan Miller, "How Not To Run An A/B Test" — evanmiller.org/how-not-to-run-an-ab-test.html. The reference article on peeking: shows with simulations how continuously monitoring an experiment inflates the real false-positive rate far above the nominal 5%, and recommends deciding the sample size in advance and looking at the result only once. In English.
• Evan Miller, Sequential Sampling Calculator (Evan's Awesome A/B Tools) — evanmiller.org/ab-testing/sequential.html. The statistically correct alternative when you genuinely need to be able to check and decide on an experiment at any time, without waiting for a fixed n — a sequential test designed for that purpose, outside this guide's scope. In English.
• Optimizely, "Statistical significance" — optimizely.com/optimization-glossary/statistical-significance. Explains how a real experimentation platform (Stats Engine) uses sequential tests precisely to allow continuous monitoring without this lesson's peeking risk. In English.