Check the pitfalls: does the result hold up to scrutiny?

Overview

Lesson 6's result is significant — p = 0.0114, though with a sample that, per lesson 5, ran slightly below the ideal (12,000 against the 12,997 the pre-registered MDE required). But "significant" isn't the same as "trustworthy," and module 7 — built in parallel to this capstone — identifies two concrete ways a significant result can still mislead. This lesson runs both: first, simpsonCheck(), to confirm the aggregate lift doesn't hide a reversal when segmented by mobile and desktop — Simpson's paradox. Second, a peeking check: what would have happened if someone had looked at the experiment's result before week 6, and why that practice, harmless as it may seem, inflates the false-positive risk far more than intuition suggests.

Connection to the module. This is the capstone's sixth layer, and the second one that draws on module 7. It implements simpsonCheck(segments) following the convention described in that module's design, and a runnable model of peeking — first over Mercado's real week-by-week experiment trace, then with a simulation that quantifies how much the look-and-stop practice inflates the false-positive rate. Neither one reveals a hidden problem in lesson 6's result — on the contrary: both put it to the test and confirm it. But the exercise of testing it, not just accepting it, is exactly what this module teaches.

An analogy: the lie detector applied even to the witness you already believed

A serious investigator doesn't stop verifying a suspect's alibi just because it already sounds convincing — they cross-check the story against the physical evidence, look for contradictions, ask "and if I look at it from another angle, does it still hold up?" Not out of groundless suspicion, but because a convincing story that also survives scrutiny is far more solid than one nobody bothered to question. Lesson 6's significant result is the convincing story. This lesson is the scrutiny: does it hold up when you look at it by segment? Does it hold up if you reconstruct how it was reached, week by week?

Worked example: Simpson, and the two faces of peeking

Part 1 — simpsonCheck(): does the lift hold up when segmented?

// simpsonCheck: compares each segment's lift against the aggregate lift, and flags
// whether any segment reverses the effect's direction relative to the aggregate --
// the signature of Simpson's paradox. Model from module 7, applied here to mobile
// vs desktop.
function simpsonCheck(segments) {
  const bySegment = segments.map((s) => {
    const rateControl = s.control.conv / s.control.n;
    const rateVariant = s.variant.conv / s.variant.n;
    const lift = (rateVariant - rateControl) / rateControl;
    return { name: s.name, rateControl, rateVariant, lift, direction: lift > 0 ? 'positive' : lift < 0 ? 'negative' : 'flat' };
  });
  const totalControl = { n: segments.reduce((a, s) => a + s.control.n, 0), conv: segments.reduce((a, s) => a + s.control.conv, 0) };
  const totalVariant = { n: segments.reduce((a, s) => a + s.variant.n, 0), conv: segments.reduce((a, s) => a + s.variant.conv, 0) };
  const aggRateControl = totalControl.conv / totalControl.n;
  const aggRateVariant = totalVariant.conv / totalVariant.n;
  const aggLift = (aggRateVariant - aggRateControl) / aggRateControl;
  const aggDirection = aggLift > 0 ? 'positive' : aggLift < 0 ? 'negative' : 'flat';
  const reversals = bySegment.filter((s) => s.direction !== aggDirection && s.direction !== 'flat');
  return { bySegment, aggregate: { rateControl: aggRateControl, rateVariant: aggRateVariant, lift: aggLift, direction: aggDirection }, reversals, paradoxDetected: reversals.length > 0 };
}

// The same 12,000 users per variant from the experiment, broken out by device -- they
// add up exactly to lesson 6's totals (384 and 456 conversions). The 40% mobile / 60%
// desktop split is the SAME one module 7's audit (lesson 8) uses on this same
// experiment -- a single source of truth for the real breakdown.
const segments = [
  { name: 'mobile', control: { n: 4800, conv: 120 }, variant: { n: 4800, conv: 144 } },
  { name: 'desktop', control: { n: 7200, conv: 264 }, variant: { n: 7200, conv: 312 } },
];

console.log('=== Part 1: simpsonCheck on mobile vs desktop ===\n');
const simpson = simpsonCheck(segments);
simpson.bySegment.forEach((s) => {
  console.log('  ' + s.name.padEnd(10) + 'control: ' + (s.rateControl * 100).toFixed(2) + '%' +
    '  variant: ' + (s.rateVariant * 100).toFixed(2) + '%' +
    '  lift: ' + (s.lift * 100).toFixed(2) + '%  (' + s.direction + ')');
});
console.log('\n  aggregate  control: ' + (simpson.aggregate.rateControl * 100).toFixed(2) + '%' +
  '  variant: ' + (simpson.aggregate.rateVariant * 100).toFixed(2) + '%' +
  '  lift: ' + (simpson.aggregate.lift * 100).toFixed(2) + '%  (' + simpson.aggregate.direction + ')');
console.log('\n  paradoxDetected = ' + simpson.paradoxDetected +
  (simpson.paradoxDetected ? '  <- ALERT' : '  <- both segments match the aggregate'));

What to expect. Part 1's output:

=== Part 1: simpsonCheck on mobile vs desktop ===

  mobile    control: 2.50%  variant: 3.00%  lift: 20.00%  (positive)
  desktop   control: 3.67%  variant: 4.33%  lift: 18.18%  (positive)

  aggregate  control: 3.20%  variant: 3.80%  lift: 18.75%  (positive)

  paradoxDetected = false  <- both segments match the aggregate

Reading Part 1. Notice that the aggregate (3.20% → 3.80%, 18.75% lift) reproduces exactly lesson 6's numbers — it's the same sum, just now broken out by segment (the same 40% mobile / 60% desktop split module 7's audit uses on this experiment). Both segments — mobile and desktop — show a positive lift, in the same direction as the aggregate: mobile improves 20.00% relative, desktop 18.18% relative. paradoxDetected: false confirms, in code, what the visual read already suggests: no segment moves in the opposite direction from the aggregate, hidden behind an average that "fixes" things by combining two different stories. recommendations' lift isn't an artifact of how the data was combined — it holds up across both channels people use to buy on Mercado.

Part 2 — The real trace, week by week: what would a peeker have seen?

Before simulating anything, it's worth looking at the experiment's real data, reconstructed week by week (the same 12,000 users per variant, accumulated over 6 weeks of ~2,000 per week), and running abTest() on each week's cumulative total — exactly what someone "peeking" at the experiment would have seen at every point:

// We reuse erf/normalCdf/abTest EXACTLY as they stood in lesson 6 of module 6 (via
// lesson 6 of this capstone), applied week by week to the experiment's real
// accumulation -- reconstructing what someone would have seen if they checked the
// result every week instead of waiting until week 6.
function erf(x) {
  const sign = x < 0 ? -1 : 1;
  x = Math.abs(x);
  const a1 = 0.254829592, a2 = -0.284496736, a3 = 1.421413741,
        a4 = -1.453152027, a5 = 1.061405429, p = 0.3275911;
  const t = 1 / (1 + p * x);
  const y = 1 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-x * x);
  return sign * y;
}
function normalCdf(x) { return 0.5 * (1 + erf(x / Math.sqrt(2))); }
function abTest({ control, variant }) {
  const { n: n1, conv: conv1 } = control, { n: n2, conv: conv2 } = variant;
  const p1 = conv1 / n1, p2 = conv2 / n2;
  const pooled = (conv1 + conv2) / (n1 + n2);
  const sePooled = Math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2));
  const z = (p2 - p1) / sePooled;
  const pValue = 2 * (1 - normalCdf(Math.abs(z)));
  return { rateControl: p1, rateVariant: p2, z, pValue, significant: pValue < 0.05 };
}

// Real week-by-week accumulation -- ends exactly at 12000/384 (control) and 12000/456
// (variant), the same totals from lesson 6.
const weeklyTrace = [
  { week: 1, controlN: 2000, controlConv: 58, variantN: 2000, variantConv: 80 },
  { week: 2, controlN: 4000, controlConv: 116, variantN: 4000, variantConv: 152 },
  { week: 3, controlN: 6000, controlConv: 180, variantN: 6000, variantConv: 228 },
  { week: 4, controlN: 8000, controlConv: 250, variantN: 8000, variantConv: 304 },
  { week: 5, controlN: 10000, controlConv: 318, variantN: 10000, variantConv: 380 },
  { week: 6, controlN: 12000, controlConv: 384, variantN: 12000, variantConv: 456 },
];

console.log('\n=== Part 2: the real trace, week by week ===\n');
weeklyTrace.forEach((w) => {
  const r = abTest({ control: { n: w.controlN, conv: w.controlConv }, variant: { n: w.variantN, conv: w.variantConv } });
  console.log('  week ' + w.week + '  n/group=' + String(w.controlN).padStart(5) +
    '  z=' + r.z.toFixed(4) + '  p=' + r.pValue.toFixed(4) +
    '  significant=' + r.significant);
});

What to expect. Part 2's output:

=== Part 2: the real trace, week by week ===

  week 1  n/group= 2000  z=1.9059  p=0.0567  significant=false
  week 2  n/group= 4000  z=2.2368  p=0.0253  significant=true
  week 3  n/group= 6000  z=2.4178  p=0.0156  significant=true
  week 4  n/group= 8000  z=2.3350  p=0.0195  significant=true
  week 5  n/group=10000  z=2.3888  p=0.0169  significant=true
  week 6  n/group=12000  z=2.5289  p=0.0114  significant=true

Reading Part 2. In week 1, with barely 2,000 users per group, the result was not significant (p = 0.0567, just above the 0.05 threshold). Someone who had looked at the experiment that first week and concluded "it's not working, let's cancel it" would have made a premature and wrong call — the real effect (which week 6 confirms) still didn't have enough sample to stand out from noise. From week 2 onward, the result becomes significant and stays that way for the rest of the experiment, though the p-value fluctuates week to week (0.0253, 0.0156, 0.0195, 0.0169, 0.0114) — it never crosses back above 0.05, but it doesn't drop in a perfectly monotonic way either. This specific case "turned out fine": no one made a wrong call based on a random spike in significance. But that was, in large part, good luck with this particular data — not a guaranteed property of the practice of looking and deciding week by week.

Part 3 — Quantifying the risk: the peeking simulation

To see why the practice of "look every week and stop at the first p < 0.05" is risky in general — beyond the fact that this time it didn't cause any harm — we simulate thousands of experiments where, by construction, there is no real effect (control and variant with the same true conversion rate), and compare two ways of deciding: looking only at the end (correct) versus looking every week and stopping at the first p < 0.05 (peeking).

// Peeking simulation: under a known NULL effect (control and variant with the same
// true rate), compares the false-positive rate of "look only at the end" against
// "look every week and stop at the first p<0.05". Uses a seeded pseudo-random
// generator (mulberry32) so the run is reproducible with node -- and approximates each
// week's conversions with a normal distribution (Box-Muller), declared as a standard
// approximation when n*p is reasonably large (here, ~64), the same as abTest()'s
// normal CDF.
function mulberry32(seed) {
  return function () {
    seed |= 0; seed = (seed + 0x6D2B79F5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
function randomNormal(rng) {
  const u1 = Math.max(rng(), 1e-12), u2 = rng();
  return Math.sqrt(-2 * Math.log(u1)) * Math.cos(2 * Math.PI * u2);
}
function simulateWeeklyConversions(rng, nPerWeek, trueRate) {
  const mean = nPerWeek * trueRate;
  const sd = Math.sqrt(nPerWeek * trueRate * (1 - trueRate));
  const draw = Math.round(mean + sd * randomNormal(rng));
  return Math.max(0, Math.min(nPerWeek, draw));
}

function peekingSimulation({ trials, weeks, nPerWeek, trueRate, alpha = 0.05, seed }) {
  const rng = mulberry32(seed);
  let peekingFalsePositives = 0;
  let fixedFalsePositives = 0;
  for (let t = 0; t < trials; t++) {
    let cumControlN = 0, cumControlConv = 0, cumVariantN = 0, cumVariantConv = 0;
    let anyWeekSignificant = false, finalWeekSignificant = false;
    for (let w = 0; w < weeks; w++) {
      cumControlN += nPerWeek; cumVariantN += nPerWeek;
      cumControlConv += simulateWeeklyConversions(rng, nPerWeek, trueRate);
      cumVariantConv += simulateWeeklyConversions(rng, nPerWeek, trueRate);
      const r = abTest({ control: { n: cumControlN, conv: cumControlConv }, variant: { n: cumVariantN, conv: cumVariantConv } });
      if (r.pValue < alpha) anyWeekSignificant = true;
      if (w === weeks - 1) finalWeekSignificant = r.pValue < alpha;
    }
    if (anyWeekSignificant) peekingFalsePositives++;
    if (finalWeekSignificant) fixedFalsePositives++;
  }
  return { trials, peekingFalseRate: peekingFalsePositives / trials, fixedFalseRate: fixedFalsePositives / trials };
}

console.log('\n=== Part 3: how much peeking inflates the false positive rate (under real H0) ===\n');
const sim = peekingSimulation({ trials: 4000, weeks: 6, nPerWeek: 2000, trueRate: 0.032, alpha: 0.05, seed: 20260801 });
console.log('trials = ' + sim.trials + '  (control and variant with the SAME true rate, 3.2%)');
console.log('fixedFalseRate   (look only at the end):        ' + (sim.fixedFalseRate * 100).toFixed(2) + '%   (nominal: 5%)');
console.log('peekingFalseRate (look every week, stop at 1st p<0.05): ' + (sim.peekingFalseRate * 100).toFixed(2) + '%');
console.log('\nInflation: ' + (sim.peekingFalseRate / sim.fixedFalseRate).toFixed(2) + 'x more false positives with peeking.');

What to expect. Part 3's output (deterministic: the fixed seed 20260801 always produces the same result):

=== Part 3: how much peeking inflates the false positive rate (under real H0) ===

trials = 4000  (control and variant with the SAME true rate, 3.2%)
fixedFalseRate   (look only at the end):        5.88%   (nominal: 5%)
peekingFalseRate (look every week, stop at 1st p<0.05): 16.80%

Inflation: 2.86x more false positives with peeking.

Reading the full result: why this matters even when "it turned out fine"

Part 3 is the piece that gives the peeking warning real weight. Under a simulated scenario where, by construction, there's no real effect between control and variant (both have exactly the same true conversion rate), the correct discipline — deciding in advance when to look at the result, and looking only once — produces a false-positive rate close to the nominal 5% (5.88% in this simulation, within the expected margin of simulation noise). But the practice of looking every week and stopping at the first p < 0.05 that shows up inflates that rate to 16.80% — almost 3 times more false positives than the alpha = 0.05 threshold promises.

The mathematical reason is simple to state, though counterintuitive: every week you look at the result is, in effect, a new opportunity for sampling noise to produce a p < 0.05 by pure chance, even when there's no real effect at all. Looking six times instead of once doesn't multiply the risk by exactly six (the looks are correlated, because each one includes the previous one's data), but it does inflate it substantially — essentially the same phenomenon as the multiple-comparisons problem.

This connects directly to Part 2: Mercado's real experiment, looked at week by week, would have shown p < 0.05 starting in week 2 and would have stayed that way for the rest of the experiment — in this particular case, looking early wouldn't have led to a wrong call, because the real effect (confirmed in week 6, with the full planned sample) turned out to be genuine. But Part 3 shows that this "luck" isn't a guarantee: if the team had repeated this same peeking practice on, say, twenty different experiments with no real effect behind any of them, nearly 1 out of 6 (16.80%, against the expected 5%) would have produced a false "variant wins" simply from looking at the wrong moment and stopping there. The discipline of deciding in advance when to look — the same protocol you already saw in module 5 — isn't a bureaucratic detail: it's what keeps the false-positive rate at the 5% alpha promises, instead of the 16.80% that looking without discipline produces.

Common mistakes

Confusing "no harm happened this time" with "peeking isn't a problem." What happens: someone reviews this lesson's Part 2, notices that looking week by week would have led to the same final conclusion (significant, ship it), and concludes that peeking, in practice, doesn't matter that much. Why it happens: Mercado's real result "turned out fine" under any looking discipline, and it's tempting to generalize from a single case to a general rule. How to spot it: if the conclusion is "nothing happened, no need to worry about this," it completely ignores Part 3 — the simulation under a real null effect, where peeking does measurably inflate the false-positive rate. How to fix it: always separate "what happened this time" (Part 2, a single experiment, with a real effect behind it) from "what happens in general" (Part 3, thousands of simulated experiments, some with a real effect and others — under H0 — with none at all). The discipline of not peeking protects against the second scenario, not the first.

Interpreting paradoxDetected: false as "there's no need to check by segment ever again." What happens: someone concludes that, since the lift held up in mobile and desktop this time, future Mercado experiments don't need to repeat the Simpson check. Why it happens: a reassuring result feels like a general rule, when it's actually the verification of one specific case. How to spot it: if a new experiment gets reported as "significant, ship it" without having run simpsonCheck() on its own segments, the discipline got dropped the moment it stopped feeling necessary. How to fix it: simpsonCheck() is a check that runs on every experiment, not a one-time confirmation inherited from a previous one — each launch has its own segment composition and its own risk of a paradox.

Treating the pitfall check as a decorative final step, after the decision is "already made." What happens: the team decides to ship recommendations the moment they see significant: true in lesson 6, and runs simpsonCheck() and the peeking check just to "complete the process," without the results of those checks being able, in principle, to change the decision. Why it happens: once a result feels "won," digging deeper feels like formality, not real analysis. How to spot it: if no one on the team can say what would have happened to the decision if paradoxDetected had come back true, the check ran without any real intention of being swayed by it. How to fix it: this lesson's pitfalls are part of the same decision process as lesson 6's significance — if a significant lift reversed when segmented, or if the result depended on having looked at exactly the right moment, lesson 8's decision would have to change, not just get noted as a curiosity.

Exercises

Exercise 1 — Design a case with a real paradox. Build a hypothetical example of two segments where the aggregate lift is positive, but both individual segments have a negative or flat lift (the classic signature of Simpson's paradox: the aggregate lies because it combines different proportions of each segment between control and variant). No need to run Node — describe the numbers with which simpsonCheck() would flag paradoxDetected: true.

See solution

One possible example: mobile has control: {n: 9000, conv: 270} (3.0%) and variant: {n: 3000, conv: 87} (2.9%) — a slightly negative lift. desktop has control: {n: 1000, conv: 50} (5.0%) and variant: {n: 7000, conv: 343} (4.9%) — also slightly negative. The paradox's trick is that variant has much more weight in desktop (7,000 of its 10,000 users) than control does (only 1,000 of its 10,000), and desktop converts better overall (~5%) than mobile (~3%). Adding it up: total control = (270+50)/10000 = 3.20%; total variant = (87+343)/10000 = 4.30% — a positive aggregate lift, even though both segments, individually, got slightly worse. The cause isn't any real effect from variant: it's that variant has more users in the segment that converts better on its own (desktop), inflating the aggregate average without the carousel having helped in any individual segment. simpsonCheck() would flag paradoxDetected: true because both segments have direction: 'negative' while the aggregate has direction: 'positive'.

Exercise 2 — Recalculate the simulation with more weeks of peeking. Without running Node yet, would you expect the peeking false-positive rate to go up or down if the team looked at the result 12 times (twice a week) instead of 6? Justify your answer with the argument from the "Reading the full result" section, and then, if you have Node handy, verify by running peekingSimulation({ trials: 4000, weeks: 12, nPerWeek: 1000, trueRate: 0.032, alpha: 0.05, seed: 20260801 }).

See solution

It should go up. Every additional look is a new opportunity for sampling noise to produce a p < 0.05 by chance — more looks, more opportunities, greater false-positive inflation, though the effect doesn't grow linearly (successive looks are correlated with each other, because they share most of the accumulated data). Running the simulation with weeks: 12 and nPerWeek: 1000 (to keep the same total accumulated users at the end) gives a peekingFalseRate higher than the 16.80% from the 6-look simulation — consistent with the intuition that peeking more frequently worsens the problem rather than diluting it.

Exercise 3 — Connect the two pitfalls to the final decision. In one or two sentences, explain why this lesson's result — paradoxDetected: false and a weekly trace that, in this case, wouldn't have led to a wrong call from peeking — gives lesson 8 a more solid foundation for its final decision than if this lesson hadn't been run at all.

See solution

Without this lesson, lesson 8's decision would rest only on a significant p-value (lesson 6) without having ruled out two concrete ways that number could be lying: that the positive aggregate effect hid a different reality by segment (Simpson), or that the process of reaching that result had been contaminated by decisions made partway through (peeking). With paradoxDetected: false and confirmation that module 5's protocol — running the full 6 weeks, without stopping early — was respected, lesson 8 can build its final decision on a result that isn't just significant, but that also survived the scrutiny this module teaches you to apply before trusting any statistical verdict.

Summary and next step

In this lesson you put lesson 6's significant result to the test from two angles the z-test, on its own, doesn't cover. simpsonCheck() confirmed the aggregate lift (18.75%) holds up in both segments — mobile (20.00%) and desktop (18.18%) — with no reversal: paradoxDetected: false. The real week-by-week trace showed that, in this particular case, looking early wouldn't have changed the final conclusion, but the simulation under a known null effect showed, with numbers (5.88% against 16.80%, a 2.86x inflation), why that discipline of not peeking matters in general, regardless of the fact that it didn't cause harm this time.

With this, the method's six layers are complete: what to measure and where (L2), retention (L3), North Star and guardrails (L4), sample size (L5), significance (L6), and pitfalls (this lesson). Lesson 8 — the final project — brings all six together into a single report, runs the complete chain in Node from start to finish, and makes the decision the whole guide has been building toward: ship, revert, or iterate.

Resources

• Evan Miller, "How Not To Run An A/B Test" — evanmiller.org/how-not-to-run-an-ab-test.html. The original and most-cited argument for why peeking inflates the false-positive rate, the conceptual basis for this lesson's Part 3. In English.
• Evan Miller, "Simple Sequential A/B Testing" — evanmiller.org/ab-testing/sequential.html. On methods that do allow looking at an experiment multiple times without inflating the false-positive rate — outside this guide's scope, but the natural next step if Mercado's team wanted to check its experiments more frequently. In English.
• Wikipedia, "Simpson's paradox" — en.wikipedia.org/wiki/Simpson's_paradox. The formal reference for the phenomenon simpsonCheck() detects, with real historical examples of the same pattern you built in Exercise 1. In English.