Module 8: Project Measure Mercados Recommendations Launch

Size the experiment: was the sample enough?

Next →

Overview

Modules 5 and 6 —and this capstone's lessons 2 through 4— worked with an already-fixed sample: 12,000 users per variant, the number the recommendations experiment ended up accumulating over its six weeks. Nobody, until now, asked whether that number was enough. This lesson asks that question backward from how it's been asked so far: instead of starting from data that already exists, you're going to calculate, with the standard sample-size formula for proportions, how many users per variant were needed to trust the result — and then compare that number against the real 12,000.

How this connects to the module. This is the full method's fourth layer, and this capstone's first one to revisit module 7 —built in parallel with this very text—. It reuses sampleSize({ baseline, mde, alpha, power }) with the same formula, unapproximated (p1*(1-p1) + p2*(1-p2), each proportion's exact variance, not 2*p*(1-p)) and the same pre-registered relative MDE, 20%, that module 7 uses in its lesson 3 and in its lesson 8's audit — not a variant of this capstone's own, the identical formula and threshold, applied over the same checkoutConversionRate from previous lessons. This lesson's result answers a question the method's lessons 5 and 6 (later in this capstone) need as a foundation: when the z-test says "significant", was it because the sample was properly sized, or because we got lucky with an insufficient sample?

An analogy: how many flips it takes to trust the coin

If you want to confirm a coin is only barely loaded —it lands heads 53% of the time, instead of the expected 50%—, flipping it 10 times isn't enough: the difference between 5 and 6 heads out of 10 flips is entirely compatible with a perfectly fair coin, pure sampling coincidence. You need far more flips —hundreds, maybe thousands— for a bias that small (just 3 percentage points) to be told apart, with confidence, from the normal noise of flipping a coin. The smaller the bias you want to detect, the more flips are needed. An A/B test's sample size obeys exactly the same logic: the smaller the effect you care about detecting (the MDE, minimum detectable effect), the more users per variant you need to trust the result.

Worked example: sampleSize() over the recommendations experiment

Before launching recommendations as an A/B test, the Mercado team had to decide how many weeks to run it — and that decision depends on how many users per variant are needed to trust the result. Following the same protocol discipline as module 5 and module 7 (lesson 2): the minimum effect that would justify keeping the carousel in production was fixed, in advance, at a relative MDE of 20% over the baseline checkout conversion rate (3.2%) — the same pre-registered threshold module 7's audit uses to calculate how much sample was needed.

// sampleSize: standard sample-size formula for two proportions, EXACTLY module
// 7's same formula (lesson 3) -- each proportion's EXACT variance,
// p1*(1-p1) + p2*(1-p2), not the 2*p*(1-p) approximation. z_alpha/2 = 1.96
// (alpha = 0.05, two-tailed) and z_beta = 0.84 (power = 0.80), the same fixed
// critical values abTest() already uses for the 95% confidence interval (module 6).
//
// n = (z_alpha/2 + z_beta)^2 * (p1*(1-p1) + p2*(1-p2)) / (p2-p1)^2   -- p2 = baseline *
// (1 + mde), the MDE expressed as a RELATIVE fraction over the baseline, module
// 7's same convention.
function sampleSize({ baseline, mde, alpha = 0.05, power = 0.8 }) {
  const zAlpha2 = 1.96; // critical z for alpha = 0.05, two-tailed
  const zBeta = 0.84;   // critical z for power = 0.80
  const p1 = baseline;
  const p2 = baseline * (1 + mde);
  const n = Math.pow(zAlpha2 + zBeta, 2) * (p1 * (1 - p1) + p2 * (1 - p2)) / Math.pow(p2 - p1, 2);
  return { requiredPerVariant: Math.ceil(n), zAlpha2, zBeta, baseline, mde, alpha, power };
}

function fmt(n) {
  return String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');
}

console.log('=== sampleSize: how many users per variant were needed ===\n');
const sizing = sampleSize({ baseline: 0.032, mde: 0.20, alpha: 0.05, power: 0.8 });
console.log('baseline (control\'s checkoutConversionRate): ' + (sizing.baseline * 100).toFixed(1) + '%');
console.log('pre-registered MDE (module 7\'s same one):     ' + (sizing.mde * 100).toFixed(0) + '% relative' +
  '  (' + (sizing.baseline * sizing.mde * 100).toFixed(2) + 'pp over the baseline)');
console.log('alpha = ' + sizing.alpha + ' (z_alpha/2 = ' + sizing.zAlpha2 + ')   power = ' + sizing.power + ' (z_beta = ' + sizing.zBeta + ')');
console.log('\nUsers required per variant: ' + fmt(sizing.requiredPerVariant));

const actualN = 12000;
console.log('\nReal sample used in the experiment (module 5): ' + fmt(actualN) + ' per variant');
const diff = actualN - sizing.requiredPerVariant;
console.log('Difference: ' + fmt(diff) + ' users per variant ' +
  (diff >= 0 ? '(the sample WAS ENOUGH)' : '(the sample fell SHORT of the ideal)'));

What to expect. When you run the file with Node, the output is exactly this:

=== sampleSize: how many users per variant were needed ===

baseline (control's checkoutConversionRate): 3.2%
pre-registered MDE (module 7's same one):     20% relative  (0.64pp over the baseline)
alpha = 0.05 (z_alpha/2 = 1.96)   power = 0.8 (z_beta = 0.84)

Users required per variant: 12,997

Real sample used in the experiment (module 5): 12,000 per variant
Difference: -997 users per variant (the sample fell SHORT of the ideal)

Reading the result: slightly short of the ideal, and why it matters

The calculation confirms, with module 7's same formula and MDE, something none of this capstone's previous lessons had verified: 12,997 users per variant were needed to detect, with 80% power, the pre-registered 20% relative MDE over a 3.2% base — and the real sample, 12,000, fell short of that number, a deficit of 997 users per variant, just under 8%. It isn't a comfortable margin: it's an experiment that ran with slightly less sample than its own design demanded.

This doesn't invalidate the result you're going to formally confirm in lesson 6 —the z-test came out significant, with p ≈ 0.0114— but it changes its weight: it isn't the kind of "significant" that arrives with a plenty-to-spare safety cushion above the minimum required, it's a result that pulled through with a narrow margin, exactly the kind of case module 7 —in its own audit of this same experiment, in that module's lesson 8— identifies as "it worked, but not from a design with margin to spare: it worked because the real effect that appeared (18.75% lift) turned out big enough to cross the significance threshold anyway, despite the deficit relative to the pre-registered MDE".

Notice, too, something the formula makes clear: if the team had wanted to detect a smaller effect, say a 10% relative MDE (half of 20%), the required sample rises to 49,718 per variant —almost four times more, not double— because the formula's denominator ((p2-p1)²) grows quadratically when the MDE gets halved. That's the mathematical reason, not just the intuitive one, behind module 7's central warning: detecting a small effect demands, disproportionately, much more sample than detecting a big one — and it's, in reverse, the same reason today's deficit (997 users, less than 8% below the ideal) is a narrow but not catastrophic margin: MDE=20% sits far enough from MDE=10% or 5% that a small deficit in n doesn't automatically ruin the ability to detect the effect, if that effect ends up being, as happened here, bigger than the minimum MDE.

Going deeper: what this formula assumes, and where it can fail

The formula used here calculates control's and variant's combined variance exactly — p1*(1-p1) + p2*(1-p2), each proportion's real variance added up, without the 2*p*(1-p) approximation that assumes p1 and p2 are almost identical (a reasonable simplification only when the MDE is very small in absolute terms). It is, with no change, module 7's same formula from its lesson 3 and its lesson 8's audit — that's why this lesson arrives at the same number, 12,997, as that audit. It also assumes the critical z values (1.96 and 0.84) correspond exactly to alpha = 0.05 (two-tailed) and power = 0.80 — the industry's conventional values, the same ones module 7 cites. Changing alpha or power would change those critical values (a more demanding test, with alpha = 0.01, would use a bigger z_alpha/2, and would ask for more sample); this lesson, following module 7's convention, keeps them fixed at their most common values, not recalculating them generally.

It's also worth saying something this lesson doesn't resolve: today's calculation uses baseline = 0.032, the control rate observed at the end of the experiment. In practice, a real team has to estimate that baseline before running the experiment, usually with historical data —like what you saw in this capstone's lesson 3, or in module 1—. If that prior estimate had been off (for example, if the team had assumed a 4% baseline instead of 3.2%), the sample size calculated in advance would have been different. Lesson 8 —the final project— revisits this idea while building the experiment's full report.

Common mistakes

Calculating the sample size after having the result, instead of before running the experiment. What happens: someone runs sampleSize() only at the end, as a retroactive "was it enough?" check, instead of calculating it before launching the experiment and using it to decide how many weeks to run it. Why it happens: it's easier to verify a number that already exists (12,000) than to commit, in advance, to a target sample without yet knowing what's going to happen. How to spot it: if nobody on the team can say what the target MDE was before the experiment started, today's calculation was done backward — as an after-the-fact justification, not a plan. How to fix it: the correct order is exactly this lesson's — define the MDE that justifies the cost (with the business team, before seeing any data), calculate sampleSize(), and use that number to decide how long to run the experiment. Verifying it afterward, as this lesson did, serves to audit the process's discipline — it doesn't replace having planned it in advance.

Treating a significant result (p < 0.05) as evidence the sample "obviously was enough". What happens: someone sees lesson 6 confirms significant: true (p ≈ 0.0114) and concludes that, since the result pulled through, the 12,000 sample "must have been enough" — without running sampleSize() to compare against the pre-registered MDE. Why it happens: a significant result feels like retroactive validation of every decision that led to it, including the sample size. How to spot it: the justification that "the n was enough" cites only the final p-value, with no backing sampleSize() calculation. How to fix it: as today's result shows, the real sample (12,000) fell short of the 12,997 the 20% MDE demanded — the result came out significant anyway, but that doesn't prove the design was correct; it's evidence that, in this specific case, the real effect was big enough to make up for a slightly insufficient sample. The only way to know whether the n was correct is comparing against sampleSize(), calculated with the pre-registered MDE — never with the final p-value as a substitute.

Choosing an arbitrarily small MDE "to be safer", without considering the sample cost. What happens: someone argues the team should have aimed to detect a 5% relative MDE, "so as not to miss any effect, however small", without calculating how much sample —or how much time— that would demand. Why it happens: a smaller MDE intuitively sounds "more rigorous", without connecting that intuition to the real cost in time and users it implies. How to spot it: if someone proposes an MDE without running sampleSize() to see how many users (or how many weeks) it implies, the proposal ignores the cost of the rigor it's asking for. How to fix it: as this lesson's example shows (MDE=10% demands almost four times MDE=20%'s sample, 49,718 versus 12,997), always run sampleSize() before committing to an MDE — the correct MDE is the smallest effect that would genuinely justify, in business terms, keeping the change, not the smallest number statistical curiosity might want.

Exercises

Exercise 1 — Recalculate with a more demanding MDE. Without running Node yet, does the required sample size go up or down if the team had demanded a 30% relative MDE instead of 20% (a more demanding decision about the minimum effect that justifies the cost, per module 7's lesson 2's criterion)? Justify your answer with the formula, then verify by running sampleSize({ baseline: 0.032, mde: 0.30, alpha: 0.05, power: 0.8 }).

See solution

It goes down. In the formula, mde determines the p2 - p1 gap in the denominator, squared — so a bigger MDE (30% instead of 20%) widens that gap, reducing the required sample size: a bigger effect is easier to detect with fewer users, exactly the opposite of what this lesson's example shows when lowering the MDE to 10%. Running sampleSize({ baseline: 0.032, mde: 0.30, alpha: 0.05, power: 0.8 }), the result gives requiredPerVariant: 6027 — less than half the 12,997 needed for the 20% MDE, and well below the real 12,000, with margin to spare.

Exercise 2 — Connect it to lesson 4's guardrail. Lesson 4's broken guardrail (checkoutLatencyP95Ms) suggests keeping recommendations has a real technical cost. If that technical cost were even bigger than it looks today, should the team raise or lower the next recommendations-related experiment's target MDE? Explain the business reasoning, not just the mathematical one.

See solution

It should raise the target MDE. If keeping and scaling recommendations costs more in infrastructure (more servers to lower latency, more engineering to optimize the recommendations engine), the minimum effect justifying that cost also rises — a 20% relative lift is no longer enough for the investment to be worth it; a lift of 30% or more might be needed to justify the extra spending. Raising the target MDE, in turn, lowers the required sample size (as Exercise 1 confirmed) — a not-always-intuitive relationship: when the business cost rises, the statistical rigor needed to justify it, measured in users needed, actually falls, because the effect that needs detecting is bigger and easier to see.

Exercise 3 — Design a new experiment's sample size. Mercado wants to test a second version of the recommendations carousel, this time based on the user's purchase history (not just the product they're viewing). The team estimates a baseline of 3.8% (the current experiment's variant conversion rate) and wants to detect a relative MDE of 10% —following module 7's same relative convention, equivalent to an absolute lift of just 0.38 percentage points over that base—, with alpha = 0.05 and power = 0.8. Calculate how many users per variant are needed.

See solution

With sampleSize({ baseline: 0.038, mde: 0.10, alpha: 0.05, power: 0.8 }): p1 = 0.038, p2 = 0.038 * 1.10 = 0.0418; the exact variance is p1*(1-p1) + p2*(1-p2) = 0.038*0.962 + 0.0418*0.9582 ≈ 0.03656 + 0.04005 ≈ 0.07661; (z_alpha/2 + z_beta)² = 2.8² = 7.84; (p2-p1)² = 0.0038² = 0.00001444. n = 7.84 * 0.07661 / 0.00001444 ≈ 41,594 users per variant. This number —more than three times the 12,000 used in the current experiment— confirms this lesson's same pattern: even though 10% relative might sound like a modest threshold, it still demands a substantially bigger sample than the original experiment's 20%, and the team would have to decide whether it's worth running the experiment longer, expanding the exposed user base, or accepting a less demanding MDE.

Summary and next step

In this lesson you calculated, with sampleSize() and the standard proportions formula —the same, unapproximated one from module 7—, that the recommendations experiment needed at least 12,997 users per variant to detect, with 80% power, the pre-registered 20% relative MDE — and confirmed the real sample (12,000 per variant) fell slightly short of that minimum, a deficit of 997 users. With this method's fourth layer, you now know that when lesson 6 confirms whether the lift is significant, that result doesn't come from a design with margin to spare: it comes from an experiment that ran below its own sample target, and that came out significant because the real effect turned out big enough — exactly the narrow margin module 7's audit identifies over this same experiment.

Where you go next. Lesson 6 brings the A/B test's full protocol (control, variant, randomization) together with the proportions z-test that finally calculates whether the +18.75% lift you've seen since lesson 2 (through the funnel) and lesson 3 (through retention) is a real effect or sampling noise.

Resources

← PreviousThe official North Star, and the guardrail that brokeNext →Run the experiment, and ask the z-test if it's real