Sample size and power

Overview

With the MDE defined (lesson 2) and alpha already known from module 6 (the false-positive risk, 0.05 by convention), only one ingredient is missing before you can calculate how many users per variant the experiment needs: power (or 1 - beta). Power is the probability that your experiment detects a real effect, given that effect truly exists and is the size of your MDE. A power of 0.80 —the industry standard, the same one you're going to use today— means: if the real effect is the size of my MDE, my experiment is going to detect it 80% of the times I run it.

Notice the word "detect" and the condition "if the real effect exists" — power is alpha's exact counterpart, but looking in the opposite direction. Alpha answers "how often does my test shout 'there's an effect!' when there actually isn't one?" (false positive). Power answers "how often does my test actually detect an effect that's truly there?" — and its complement, beta (beta = 1 - power), is the probability of the opposite error: a false negative, not detecting an effect that does exist. With alpha, power, and lesson 2's MDE, you finally have the sample-size formula's three full ingredients — the number this lesson builds and runs.

How this connects to the module. This lesson builds sampleSize(), this module's central function, reused unchanged in lesson 8's mini-project to size the recommendations A/B test from scratch. The alpha=0.05 this function uses is exactly the same alpha module 6 used to calculate abTest()'s p-value — two faces of the same decision: how willing you are to be wrong declaring an effect that doesn't exist.

An analogy: the pregnancy test and its two ways of failing

A pregnancy test can fail in two completely different ways. It can come out positive when there's actually no pregnancy —a false positive— or it can come out negative when there actually is one —a false negative—. No maker of these tests promises zero errors of any kind; instead, they publish two numbers: how often each type of error happens. A test with very low alpha almost never comes out positive by mistake — but if it also has low power, it also misses real pregnancies too often, coming out negative when it shouldn't. A well-designed test aims for both things at once: few false positives (low alpha) and a high ability to detect what's real when it's there (high power) — and achieving both simultaneously, as you're about to see today, is exactly what demands a big enough sample.

An A/B test has the same two-error structure. Alpha (you already know it from module 6) is the probability that the z-test says "recommendations has an effect" when it actually has none — the false positive. Beta is the probability that the z-test says "there's no evidence of any effect" when recommendations is truly generating the lift you were looking for — the false negative, the exact equivalent of a pregnancy test that misses a real pregnancy. Power (1 - beta) is the probability of avoiding that second error: detecting the effect when it's truly there. And here's the piece that connects everything to the previous lesson's MDE: the smaller the real effect you're trying to detect, the more it resembles —just like a pregnancy barely begun, barely detectable— background noise, and the more "sensitive" (more sample) your test needs to not let it slip by.

Worked example: sampleSize() over recommendations

The standard sample-size formula for comparing two proportions —the same one calculators like Evan Miller's or Optimizely's use— is this:

n_per_variant = ((z_alpha/2 + z_beta)^2 * (p1*(1-p1) + p2*(1-p2))) / (p2-p1)^2

Where p1 is the baseline (the control rate), p2 = p1 * (1 + MDE) (the rate you want to be able to tell apart from p1), z_alpha/2 = 1.96 for a two-tailed alpha of 0.05 (the same critical value from module 6's z-test), and z_beta = 0.84 for a power of 0.80. The numerator combines how much confidence you demand (z_alpha/2 and z_beta, bigger if you want more rigor) with how much natural "noise" is in the data (p1*(1-p1) + p2*(1-p2), the two proportions' combined variance). The denominator is the gap between p1 and p2 squared — and that square, as you're about to see in today's output, is the exact mathematical reason a small MDE demands so much sample.

// sampleSize: standard sample-size formula for two independent proportions
// (control vs variant), with no external dependencies.
// z_alpha/2 = 1.96 (alpha=0.05, two-tailed) and z_beta = 0.84 (power=0.80) are
// the standard critical values of the normal distribution for these conventional
// thresholds -- the same alpha=0.05 from module 6's z-test.
function sampleSize({ baseline, mde, alpha = 0.05, power = 0.80 }) {
  const zAlpha2 = 1.96;
  const zBeta = 0.84;
  const p1 = baseline;
  const p2 = baseline * (1 + mde);
  const numerator = Math.pow(zAlpha2 + zBeta, 2) * (p1 * (1 - p1) + p2 * (1 - p2));
  const denominator = Math.pow(p2 - p1, 2);
  return Math.ceil(numerator / denominator); // ALWAYS round up
}

function fmt(n) {
  return String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');
}

// Pedagogical data: Mercado's checkoutConversionRate, baseline=3.2% (the
// control rate, identical to module 5). We test the same MDEs from
// lesson 2's table.
const baseline = 0.032;
console.log('=== sampleSize() over recommendations, baseline=3.2%, alpha=0.05, power=0.80 ===\n');
[0.20, 0.10, 0.05].forEach((mde) => {
  const n = sampleSize({ baseline, mde });
  const p2 = baseline * (1 + mde);
  console.log('MDE=' + (mde * 100).toFixed(0).padStart(2) + '%  ->  p2=' + (p2 * 100).toFixed(3) +
    '%  ->  n per variant = ' + fmt(n));
});

What to expect. When you run the file with Node, the output is exactly this:

=== sampleSize() over recommendations, baseline=3.2%, alpha=0.05, power=0.80 ===

MDE=20%  ->  p2=3.840%  ->  n per variant = 12,997
MDE=10%  ->  p2=3.520%  ->  n per variant = 49,718
MDE= 5%  ->  p2=3.360%  ->  n per variant = 194,307

There it is, with concrete numbers, the relationship lesson 1 previewed with the barely-loaded coin: going from a 20% MDE to a 5% one —four times smaller—, the sample size needed doesn't quadruple: it multiplies by nearly 15, from 12,997 to 194,307 users per variant. That explosion isn't an accident of these specific numbers — it's written directly into the formula: since the denominator is (p2-p1) squared, when you halve the gap between p1 and p2, the denominator shrinks to a quarter, and the full result (numerator divided by denominator) multiplies, roughly, by four. Reducing the MDE doesn't cost linearly more sample — it costs quadratically more.

Verifying the formula against a known case

Before trusting any implementation of a statistical formula, it's worth confirming the order of magnitude makes sense. We run sampleSize() over a second, higher baseline (5%, a typical e-commerce conversion value), with two different MDEs:

console.log('\n=== Verification: baseline=5%, a different reference case ===\n');
[0.20, 0.10].forEach((mde) => {
  const n = sampleSize({ baseline: 0.05, mde });
  console.log('baseline=5%  MDE=' + (mde * 100).toFixed(0) + '%  ->  n per variant = ' + fmt(n));
});

What to expect.

=== Verification: baseline=5%, a different reference case ===

baseline=5%  MDE=20%  ->  n per variant = 8,146
baseline=5%  MDE=10%  ->  n per variant = 31,196

Both numbers land exactly where statistical intuition says they should: thousands to tens of thousands of users per variant to detect relatively small lifts (10-20%) over a single-digit baseline — the same order of magnitude public sample-size calculators, like Evan Miller's or Optimizely's, report for comparable conversion scenarios. If sampleSize() had returned, say, tens instead of thousands, or millions instead of thousands, that would be a clear sign of an implementation error — the formula is calibrated to produce numbers in this range for single-digit conversions and double-digit MDEs, and confirming that, before trusting the result over real data, is exactly the verification discipline you already saw with abTest() in module 6.

Common mistakes

Deciding the experiment's duration "by feel", without having calculated the sample size first. What happens: a team decides to run an experiment "two weeks, to have time to see something" or "until it feels like enough", without having first calculated how many users per variant the MDE they care about needs. Why it happens: setting a calendar duration (weeks) feels more natural and easier to plan than setting a number of users, especially before knowing this lesson's formula. How to spot it: nobody on the team can cite the target n per variant, only the date on which "we'll check the result". How to fix it: always calculate sampleSize() first, with baseline, MDE, alpha, and power defined in advance — and only afterward translate that n into an estimated duration, dividing by the expected weekly traffic per variant. Duration is a consequence of the sample size, never the starting point.

Rounding sampleSize()'s result down, "to save time". What happens: sampleSize() returns, say, 12,997 users per variant, and someone decides to round to 12,000 "to keep it simple" or because the available traffic falls a bit short. Why it happens: the difference seems minimal (less than 8%), and cutting a few days off the end of the experiment's wait feels like a reasonable win. How to spot it: the real n used in the experiment is smaller than the n sampleSize() calculated for the declared MDE and power. How to fix it: today's code's Math.ceil() isn't an implementation detail — it's the guarantee that the promised power (80% probability of detecting the effect, if it exists) holds. Rounding down reduces the real power below the declared 80%, with nobody having explicitly decided that — exactly the kind of silent shortcut that makes an experiment "barely scrape by" instead of being properly sized.

Confusing z_alpha/2 with z_beta, or using a single term instead of the sum of both. What happens: someone reimplements the formula using only z_alpha/2 (1.96) in the numerator, forgetting to add z_beta (0.84) — an easy mistake to make because alpha is already familiar from module 6, while z_beta is new in this lesson. Why it happens: the significance z-test's formula (module 6) uses a single critical value (z_alpha/2), and it's easy to assume, out of habit, sample size works the same way. How to spot it: the calculated n consistently comes out smaller than expected — omitting z_beta shrinks the term (z_alpha/2 + z_beta)^2 from 2.8^2 = 7.84 to just 1.96^2 = 3.84, almost half. How to fix it: remember sample size depends on two different risks —the false positive (alpha, already known) and the false negative (beta, new today)— and the formula needs both critical values added together, exactly as in today's sampleSize(): Math.pow(zAlpha2 + zBeta, 2).

Exercises

Exercise 1 — Calculate the impact of raising power. Without running Node yet, predict: if you keep baseline=3.2% and MDE=20%, but raise power from 0.80 to 0.90 (which raises z_beta from 0.84 to roughly 1.28), does the needed n go up or down? Then, modify sampleSize() to accept that zBeta as a parameter and confirm your prediction by running the code.

See solution

The needed n goes up. A higher power (0.90 instead of 0.80) demands a higher probability of detecting the real effect if it exists, which always requires more sample, never less — it's an additional demand on the same test, not a relaxation. Modifying sampleSize() to use zBeta=1.28 instead of 0.84, the term (zAlpha2 + zBeta)^2 rises from 2.8^2=7.84 to 3.24^2=10.4976 — an increase of roughly 34%, which directly shows up as an n about 34% bigger (from 12,997 to roughly 17,400). The general lesson: any additional demand for rigor —lower alpha, higher power, smaller MDE— always gets paid for with more sample, never the other way around.

Exercise 2 — Translate n into weeks of experiment. If Mercado's checkoutConversionRate receives, on average, 4,000 new users a week split evenly between control and variant (2,000 per variant per week), how many weeks would the experiment need to reach today's MDE=20% n=12,997 per variant?

See solution

12,997 / 2,000 ≈ 6.5 weeks, rounded up to 7 weeks (never round the duration down, for the same reason as today's second common mistake: it would leave you below the target n and the promised power). Notice this is very close to the real 6 weeks the recommendations experiment used in modules 5 and 6 — a coincidence lesson 8's mini-project is going to explore in more detail.

Exercise 3 — Decide between three options with limited traffic. Mercado only has 3,000 new users a week available for a new experiment (1,500 per variant per week), and the team wants to finish it in at most 8 weeks (12,000 users per variant total). With baseline=3.2%, alpha=0.05, and power=0.80, what's the smallest MDE that experiment can afford to detect, using today's table as reference? What would you do if the team insists on being able to detect a 5% MDE?

See solution

With 12,000 users per variant available over 8 weeks, the experiment can detect, with power=0.80, a 20% MDE (which needs 12,997 — very close to the available limit) but it cannot detect a 10% MDE (which needs 49,718), let alone 5% (194,307) — both demand far more sample than the traffic allows in 8 weeks. If the team insists on a 5% MDE, the real options are: (a) drastically extend the experiment's duration —194,307 users per variant, at 1,500 a week, would take more than 129 weeks, almost two and a half years, clearly unfeasible—; (b) accept a lower power (less reliable, risking not detecting the effect even if it exists); or (c), the practically correct answer, accept that with this traffic the experiment can only be honestly sized for a bigger MDE, and reserve detecting smaller effects for when the product has more traffic. Pretending a 5% MDE can be detected with 12,000 users would be exactly the kind of shortcut this lesson's second common mistake warns about.

Summary and next step

In this lesson you completed the sample-size formula's three ingredients —MDE (lesson 2), alpha (already known from module 6), and power— and built sampleSize(), run over Mercado's checkoutConversionRate: 12,997 users per variant for a 20% MDE, 49,718 for 10%, and 194,307 for 5%. You saw, with real numbers, why that relationship isn't linear but quadratic —the formula's (p2-p1)^2 denominator— and verified the implementation's order of magnitude against a second reference case (baseline=5%).

Before moving on you should be able to: explain the difference between alpha and power (the two types of error an experiment can make); calculate sampleSize() by hand given baseline, MDE, and the standard critical values; and explain why roughly halving the MDE quadruples the needed sample.

With this the module's first half ends — the "how much sample do I need" part. The second half, lessons 4 through 7, switches questions: even if you have exactly the right sample, what can ruin your result anyway? Lesson 4 opens that catalog with the first pitfall, and probably the most common: peeking — the risk of looking at the experiment before it's over, and stopping the moment you see something promising.

Resources

• Evan Miller, Sample Size Calculator (Evan's Awesome A/B Tools) — evanmiller.org/ab-testing/sample-size.html. An interactive calculator for the same formula implemented today — useful for quickly exploring how n changes as you move baseline, MDE, alpha, or power independently. In English.
• Optimizely, "Sample size calculations for A/B tests and experiments" — optimizely.com/insights/blog/sample-size-calculations-for-experiments. The same standard proportions formula, explained from the perspective of an experimentation platform used in production by thousands of teams. In English.
• Wikipedia, "Power (statistics)" — en.wikipedia.org/wiki/Power_(statistics). The formal reference for power, beta, and their relationship to sample size and effect size — the full theoretical basis behind sampleSize(). In English.