Module 7: Sample Size And Pitfalls

Module introduction: from "the lift is real" to "how much sample did I need, and what pitfalls could have fooled me"

Next →

Why this module

Module 5 rigorously designed the recommendations experiment: control and variant randomly assigned, the null hypothesis formulated in advance, and the lift calculated over the launch's real numbers — controlRate = 3.20%, variantRate = 3.80%, lift = +18.75%. Module 6 took those exact same numbers and applied a real proportions z-test to them: it calculated the z-score, the p-value with a normal CDF approximation, and the 95% confidence interval — and with that, you were finally able to answer the question left open since module 5's lesson 6: is that +18.75% a real effect from recommendations, or is it the kind of variation that would show up all the same, by pure coincidence, even if the carousel did absolutely nothing? With z ≈ 2.53 and a pValue ≈ 0.011 — below the conventional 0.05 threshold — module 6's answer was: probably real, not chance.

It would be reasonable to think the work ends there. It doesn't. Module 6's p-value answers a very specific question —is this difference too big to be coincidence, given the sample size actually used?— but two equally important questions were left completely out, and both are about what happened before and around that calculation:

  1. How did you know, before launching the experiment, that 12,000 users per variant were going to be enough? The Mercado team ran the experiment with that sample size, but where did that number come from? If recommendations's real effect had been smaller —say, a 5% lift instead of the 18.75% that ended up appearing— would that same 12,000 users per variant have been enough to detect it, or would the experiment have been doomed, by design, to never reach significance no matter how long it ran?
  2. Was the process that produced that 0.011 p-value clean? A trustworthy p-value silently assumes several things: that nobody looked at the result midway and decided to stop the experiment the moment something promising appeared, that checkoutConversionRate was the only metric being tested (not one of twenty), that the lift holds up if you split traffic by device type, and that this +18.75% isn't just users' initial curiosity about a new carousel, about to fizzle out in the coming weeks.

This module answers those two questions. The first —how much sample was needed— is arithmetic: a standard formula that translates three decisions (MDE, alpha, power) into a concrete number of users per variant, calculated before running the experiment, not after. The second —whether the process was clean— is a catalog of five concrete pitfalls, each with its own lesson, that can make you believe an effect is real when it isn't (or hide a real one that is).

Modules 5-6 (already covered)            Module 7 (here)
──────────────────────────               ─────────────────────────────
Designing the experiment +               How much sample was needed
measuring whether the lift               BEFORE running, to be able to
is real (z-test, p-value, CI)            trust the result?

"The +18.75% lift looks real             What pitfalls could the team
 (p ≈ 0.011)"                            have fallen into without
                                          realizing -- peeking, multiple
                                          comparisons, Simpson, novelty,
                                          p-hacking?

An analogy: the barely-loaded coin

Imagine you have two coins that look identical. One is perfectly fair: 50% heads, 50% tails. The other is barely loaded: 52% heads, 48% tails — a real difference, but a small one. If you flip each coin only 10 times, there's practically no way to tell them apart: the natural noise of 10 flips —getting 6 heads out of 10, or even 7, happens often even to a perfectly fair coin— is much bigger than the real 2% difference you're trying to detect. You could flip the loaded coin 10 times and get 4 heads, a result that at a glance seems to suggest it's loaded the other way.

To confidently tell a coin loaded at 52% apart from one fair at 50%, you need many more flips — hundreds, sometimes thousands, depending on how sure you want to be and how small the difference you insist on being able to detect is. And here's the point that organizes this whole module: if instead you wanted to tell a heavily loaded coin (80% heads) apart from a fair one, a few flips would already tell you clearly — the difference is so big the noise can't hide it. The smaller the effect you want to be able to detect, the vastly more sample you need. That's, in one sentence, this module's lesson 3's central relationship, and the exact reason an experiment with a small lift and insufficient sample can fail to detect something that is real — not because the effect doesn't exist, but because you never gave the experiment enough "flips" to see it clearly.

The map of this module

Eight lessons, each solving a specific piece of "how much sample do I need" and "where could I go wrong without realizing it":

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) What's missing after designing (M5) and measuring significance (M6)?
L2        What exactly is the MDE, and why is choosing it a business decision?
L3        How many users per variant do I need, given MDE, alpha, and power?
L4        Why is watching the test constantly and stopping at the first p<0.05 a trap?
L5        Why does testing 20 metrics at once almost guarantee a false "winner"?
L6        How can an aggregate lift lie if I don't check the segments?
L7        What is the novelty effect, and what is "cooking" the analysis (p-hacking)?
L8        Project: size the recommendations A/B and audit its pitfalls

Lesson 2 starts with the formula's first ingredient: the MDE (minimum detectable effect, the smallest effect you actually care about being able to detect) — and why choosing it isn't a calculation, it's a business decision disguised as a number. Lesson 3 brings the MDE together with alpha (the false-positive risk you already know from module 6) and power (the probability of detecting a real effect, if one exists) into the full sample-size formula, implemented in sampleSize() and run over Mercado — there you're going to see, with real numbers, why a small lift demands a huge sample. Lessons 4 through 7 switch topics: it's no longer about how much sample you need, but about what can ruin a result even with the correct sample — looking at the test too early (lesson 4), testing too many metrics at once (lesson 5), not segmenting and getting bitten by a hidden reversal (lesson 6), and confusing a passing bump with a permanent effect, or worse, forcing the analysis until something comes out significant (lesson 7). Lesson 8, the mini-project, has you do both things end to end over the real recommendations launch: calculating the sample size the experiment needed from the start, and auditing module 5's result looking for exactly these five pitfalls.

Worked example: where we come from, where we're going

Before diving into the first concept, let's see in code what we inherited from modules 5 and 6, and the two exact questions this module opens:

// Recap of what Mercado knows up through module 6, and the two exact
// questions module 7 starts to answer.
const whatWeKnowSoFar = {
  module5: {
    topic: 'A/B testing fundamentals (design)',
    control: '3.20% (n=12000)',
    variant: '3.80% (n=12000)',
    lift: '+18.75%',
  },
  module6: {
    topic: 'statistical significance',
    zScore: '~2.53',
    pValue: '~0.011 (< 0.05)',
    conclusion: 'the lift looks real, not chance -- over THIS sample size',
  },
  module7: {
    topic: 'sample size and pitfalls',
    openQuestions: [
      'How much sample was needed BEFORE running, to be able to trust the result?',
      'What pitfalls could the team have fallen into without realizing it (peeking, etc)?',
    ],
  },
};

console.log('=== Where we come from, where we\'re going ===\n');
console.log('Module 5 (design): control=' + whatWeKnowSoFar.module5.control +
  '  variant=' + whatWeKnowSoFar.module5.variant + '  lift=' + whatWeKnowSoFar.module5.lift);
console.log('Module 6 (significance): z=' + whatWeKnowSoFar.module6.zScore +
  '  pValue=' + whatWeKnowSoFar.module6.pValue);
console.log('  Module 6\'s conclusion: ' + whatWeKnowSoFar.module6.conclusion);
console.log('\nModule 7 (here) -- two questions module 6 did NOT answer:');
whatWeKnowSoFar.module7.openQuestions.forEach((q, i) => console.log('  ' + (i + 1) + '. ' + q));

What to expect. When you run the file with Node, the output is exactly this:

=== Where we come from, where we're going ===

Module 5 (design): control=3.20% (n=12000)  variant=3.80% (n=12000)  lift=+18.75%
Module 6 (significance): z=~2.53  pValue=~0.011 (< 0.05)
  Module 6's conclusion: the lift looks real, not chance -- over THIS sample size

Module 7 (here) -- two questions module 6 did NOT answer:
  1. How much sample was needed BEFORE running, to be able to trust the result?
  2. What pitfalls could the team have fallen into without realizing it (peeking, etc)?

Notice the phrase "over THIS sample size" in module 6's conclusion — it isn't a minor detail. The 0.011 p-value is a conditional claim: given that you used 12,000 users per variant, the observed difference is unlikely under the null hypothesis. But that claim doesn't tell you whether 12,000 was the right number to begin with, nor whether the path to that p-value was free of the pitfalls this module is about to catalog. Both are exactly what's missing.

The boundary: what does NOT belong in this module

Two limits matter so you don't get confused about what you already covered:

  • The z-test, the p-value, and the confidence interval — how they're calculated and what they mean — you already built those in module 6, and this module doesn't teach them again. Here they're taken as known: you're going to use them as reference (for example, in lesson 8's mini-project), but abTest()'s mechanics don't change.
  • Gradual rollout, feature flags, canary release — the mechanics of shipping the winning variant to 100% of users once you decide to ship — is content for shipping-and-iterating-products-guide, not this guide. Here you stay at the moment of deciding with evidence, before that next step.

Within this module: sizing the sample before running an experiment, and auditing an already-obtained result looking for the five pitfalls that can make it look more —or less— trustworthy than it really is.

Common mistakes

Treating module 6's p-value as the end of the story, without asking how it was reached. What happens: a team sees pValue ≈ 0.011 < 0.05 and considers the "is it real?" discussion closed, without asking whether the sample size was calculated in advance or whether anyone looked at the result before the 6 weeks were up. Why it happens: a p-value is a concrete, definitive number — it looks like the final answer, and after all the work in modules 5 and 6 it's tempting to treat it as such. How to spot it: nobody in the conversation can describe how many times the dashboard was checked during the experiment, nor whether the sample size was decided in advance or adjusted along the way. How to fix it: as you'll see in lessons 3 through 7, a trustworthy p-value depends on decisions made before looking at the data (the sample size) and on discipline during the experiment (not peeking, not testing too much). The number alone, without that context, isn't enough.

Assuming "more sample is always better" without understanding the trade-off's cost. What happens: someone proposes simply running every experiment with the biggest possible sample "to be safe", without calculating how much is actually needed for the effect they care about. Why it happens: intuitively, more data feels more trustworthy, and calculating the exact sample size seems like a dispensable extra step when "you already have plenty of users available". How to spot it: an experiment runs for months with nobody able to explain why that specific duration, or what MDE was being targeted. How to fix it: as you'll see in lesson 3, the correct sample size depends on an explicit decision —the MDE— not on "the more, the better". Running too long wastes time and exposes the product to a possibly worse variant for longer than needed; running too little dooms you to never detecting the effect you were looking for.

Thinking this module's five pitfalls are bad-faith mistakes. What happens: someone assumes peeking, multiple comparisons, Simpson's paradox, the novelty effect, and p-hacking are things only dishonest teams do, deliberately trying to inflate a result. Why it happens: the name "p-hacking" sounds like fraud, and it's easy to picture these pitfalls as deliberate manipulation instead of what they almost always are: reasonable habits —checking a dashboard, measuring several metrics at once, getting excited about an early result— that nobody flagged as risky. How to spot it: the conversation about these pitfalls focuses on "who cheated" instead of "what process let this happen without anyone noticing". How to fix it: as you'll see in lessons 4 through 7, each of these pitfalls is easy to fall into in good faith, precisely because it feels like common sense (checking the result often, measuring everything you can) until you understand the math behind it. This module's goal isn't to suspect anyone — it's to recognize the pattern before falling into it.

Exercises

Exercise 1 — Sort the question into the right module. For each question about the recommendations experiment, say whether you already answered it in modules 5-6, or whether it's exactly what starts in this module (7):

  • (a) "Is the +18.75% lift statistically significant, given we used 12,000 users per variant?"
  • (b) "Was 12,000 users per variant enough to detect an effect of the size we expected, before running the experiment?"
  • (c) "Did anyone on the team check the experiment's dashboard before the planned 6 weeks were up?"
  • (d) "What is control and what is variant in this experiment?"
See solution
  • (a) Modules 5-6 (already answered). The significance calculation —z-score, p-value— is exactly module 6's job, already done over these numbers.
  • (b) Module 7 (here). The sample size needed, calculated before running, is this module's lesson 3.
  • (c) Module 7 (here). Whether the team looked at the result too early is exactly the peeking question, lesson 4.
  • (d) Module 5 (already answered). Defining control and variant is module 5's lesson 3 content.

Exercise 2 — Explain the analogy in your own words. In 2-3 sentences, explain why, to tell a coin loaded at 52% apart from a fair coin at 50%, you need many more flips than to tell an 80%-loaded coin apart from a fair one.

See solution

The smaller the real difference between the two coins (2 percentage points versus 30), the more its behavior resembles the natural noise of a short flip series —both coins can, by pure chance, give very similar results in a few attempts—. For that small difference to become visible above the noise, you need to accumulate many more flips, so each coin's average converges to its true probability and the difference between the two stops being explainable by chance. A big difference (80% versus 50%), on the other hand, is already visible with few flips because normal noise can't hide such a wide gap.

Exercise 3 — Predict the rest of the module. Based on the map of the eight lessons, in which lesson would you expect to find the explanation for why a lift that looks positive in aggregate across all traffic could actually be hiding the fact that it got worse for mobile users? Justify your answer with the question that lesson answers.

See solution

Lesson 6 (Simpson's paradox). The question that lesson answers, per the map, is "how can an aggregate lift lie if I don't check the segments?" — exactly the described scenario: a positive aggregate result hiding a reversal when you split traffic by a confounder, like device type.

Summary and next step

In this lesson you saw the exact boundary between what you already know —designing an experiment (module 5) and calculating whether its result is significant (module 6)— and what this module adds: how much sample was needed from the start, and what specific pitfalls the team could have fallen into without realizing it along the way. You saw the analogy that's going to run through the first half of the module —the barely-loaded coin, and why a small effect demands vastly more "flips" than a big one— and the full map of the eight lessons that build, piece by piece, both the sample-size calculation and the pitfall catalog.

Before moving on you should be able to: explain in your own words why a significant p-value alone isn't enough to trust an experiment, and broadly locate which lesson in this module solves which pending piece.

Lesson 2 starts with the sample-size formula's first ingredient: what exactly is the MDE, and why is choosing it, underneath, a business decision and not a statistical calculation?

Resources

  • Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — experimentguide.com. The reference book for this whole part of the guide (modules 5 through 7); the chapter on experiment planning covers this module's sample size and pitfalls in depth. In English.
  • Evan Miller, "How Not To Run An A/B Test" — evanmiller.org/how-not-to-run-an-ab-test.html. The classic article on why checking an experiment often and stopping at the first promising result inflates the false-positive rate far beyond what you'd think — lesson 4's central topic. In English.
  • Wikipedia, "Power (statistics)" — en.wikipedia.org/wiki/Power_(statistics). The formal reference for power (1-beta), one of the three ingredients of the sample-size formula you're going to implement in lesson 3. In English.
← PreviousProject: is the recommendations launch real?Next →The minimum detectable effect (MDE)