Could the lift be pure coincidence?

Overview

Before touching a single formal statistics formula, this lesson does something simpler and, underneath, more convincing: look at Mercado's own control group, day by day, with no experiment running yet. If a group nobody changed anything for already bounces from one day to the next just from the randomness of which users showed up that day, then a difference between two different groups (control versus variant) has to clear that background noise before it can be called a "real effect". This lesson measures exactly that background noise, with Mercado's real weekly numbers.

How this connects to the module. Lesson 1 laid out the module's central question: is the +18.75% lift real or is it chance? This lesson doesn't answer it yet —that starts in lesson 3, with the null hypothesis and the p-value— but it builds the intuition that makes that answer make sense: you're going to see, with simple arithmetic, that sampling noise alone already produces swings the size of the lift that so excited the team. Without this lesson, lesson 3's p-value would just be a memorized formula; with it, it's the natural answer to something you've already seen with your own eyes.

An everyday analogy: the party that fills up differently every week

Picture a bar that runs the exact same promotion every Friday: two drinks for the price of one, 8 to 10 pm. The owner changes nothing from one Friday to the next — same hours, same price, same sign on the door. And yet, one Friday 140 people show up, the next 165, the next 148, the next 190. The owner didn't do anything different: what changed was who decided to go out that particular night — the weather, whether there was another event in town, whether people got paid Friday or Monday. That back-and-forth, Friday to Friday, says nothing about whether the promotion "improved" or "got worse" — it's simply the natural variation of counting people in a small sample of the city's total population.

Now imagine that, one Friday, the owner adds a new DJ and 190 people show up — a big jump compared to the previous Friday (148). Did the DJ work, or was it just a Friday with good weather and nothing else? You can't know from that number alone — you need to compare that jump against how much attendance normally bounces, with nothing changed. If the bar already bounces between 140 and 190 with no changes at all, a jump to 190 with a new DJ doesn't prove much. If the bar normally moves between 145 and 155, a jump to 190 does stand out. Mercado's control group is that bar with no DJ: by measuring how much it bounces on its own, you learn how big a "jump" needs to be before it stops looking like noise and starts looking like a real signal.

Worked example: control's conversion, day by day, with no experiment at all

Module 5 reported control's weekly aggregate: 384 conversions out of 12,000 visits, 3.2%. That number is a seven-day average. Split day by day —the same people, the same checkout, absolutely nothing different between Monday and Sunday—, it looks like this:

// Day-to-day variation WITHIN the same control group -- with no real change -- to
// show that sampling noise alone already produces big swings. The 7 days are a real
// partition of module 5's same 384/12000 (nothing invented in the weekly aggregate),
// just split day by day to see the variation.
const weeklyRate = 384 / 12000; // 3.2%, the same aggregate from module 5

const days = [
  { day: 'Mon', n: 1716, conv: 48 },
  { day: 'Tue', n: 1714, conv: 60 },
  { day: 'Wed', n: 1714, conv: 50 },
  { day: 'Thu', n: 1714, conv: 54 },
  { day: 'Fri', n: 1714, conv: 58 },
  { day: 'Sat', n: 1714, conv: 66 },
  { day: 'Sun', n: 1714, conv: 48 },
];

console.log('=== Day-to-day variation within the SAME control (nothing changed) ===\n');
let totalN = 0, totalConv = 0;
days.forEach((d) => {
  const rate = d.conv / d.n;
  const liftVsWeekly = (rate - weeklyRate) / weeklyRate;
  totalN += d.n;
  totalConv += d.conv;
  console.log(
    '  ' + d.day + '  n=' + d.n + '  conv=' + String(d.conv).padStart(2) +
    '  rate=' + (rate * 100).toFixed(2) + '%' +
    '  vs weekly average: ' + (liftVsWeekly >= 0 ? '+' : '') + (liftVsWeekly * 100).toFixed(1) + '%'
  );
});
console.log('\nWeek total: ' + totalConv + '/' + totalN + ' = ' + ((totalConv / totalN) * 100).toFixed(2) + '%  (matches module 5\'s 3.2%)');

What to expect. When you run the file with Node, the output is exactly this:

=== Day-to-day variation within the SAME control (nothing changed) ===

  Mon  n=1716  conv=48  rate=2.80%  vs weekly average: -12.6%
  Tue  n=1714  conv=60  rate=3.50%  vs weekly average: +9.4%
  Wed  n=1714  conv=50  rate=2.92%  vs weekly average: -8.8%
  Thu  n=1714  conv=54  rate=3.15%  vs weekly average: -1.5%
  Fri  n=1714  conv=58  rate=3.38%  vs weekly average: +5.7%
  Sat  n=1714  conv=66  rate=3.85%  vs weekly average: +20.3%
  Sun  n=1714  conv=48  rate=2.80%  vs weekly average: -12.5%

Week total: 384/12000 = 3.20%  (matches module 5's 3.2%)

Look closely at Saturday: +20.3% relative to the weekly average. Bigger than the +18.75% lift the team saw between control and variant in module 5 — and that +20.3% on Saturday happened with no experiment, no change, no new recommendation at all. It was, purely, the bar with no DJ, one Friday with more people than usual. If control alone can already jump 20.3% from one day to the next with no real cause, the uncomfortable question becomes unavoidable: is the +18.75% between control and variant truly recommendations's effect, or is it the same kind of bounce you just saw on Saturday, just spread across two different groups instead of two different days?

Why a small sample bounces more

The bounce you just saw isn't a flaw in Mercado's data — it's a mathematical property of counting over a sample instead of over the whole population. The fewer people that go into a rate's calculation, the more each individual person weighs on the final result. With ~1,714 visits a day, a difference of just 10 or 12 people who decided to buy (or not) moves the observed rate several tenths of a percentage point — and those tenths, over a 3.2% base, already represent a double-digit relative jump. If instead it were measured with a million visits a day, that same individual variation of "10 or 12 people" would be diluted to something almost invisible, because there are many, many more people whose decisions average each other out.

This is the deep reason behind a phrase you're going to hear often in experimentation: "with a small sample, anything can look like an effect". It's not that the math is wrong — it's that sampling noise, with a small sample, is simply bigger, and it takes an even bigger difference to tell it apart from noise. That idea —how much sample is needed to "see through" the noise— is exactly module 7's topic (sampleSize); for now, all you need is the intuition that the noise exists, is real, and you've already seen it with your own numbers.

Common mistakes

Concluding "control is unstable, something's wrong with the data". What happens: someone sees the day-by-day table and assumes there's an instrumentation bug, because "conversion shouldn't move that much for no reason". Why it happens: intuition expects a "real" rate to be a fixed number, and any movement gets interpreted as an error, not as expected variation. How to spot it: if the explanation offered is "we need to check the tracking" instead of "this is normal sampling noise", the variation is being misread as a technical problem. How to fix it: a rate calculated over a sample always has variation — it's not a bug, it's the nature of counting over a subset. The right question isn't "why does it move?", it's "does it move more than chance would explain?" — and that formal question is exactly lessons 3 and 6's.

Using the most extreme day's bounce to rule out ANY future lift. What happens: someone sees Saturday's +20.3% and concludes "so no lift below 20% means anything, ever", applying that figure as a universal threshold. Why it happens: a single dramatic number sticks and gets over-generalized, as if it were a fixed rule instead of one specific week's isolated observation. How to spot it: if someone cites "20%" as a significance threshold without having run any formal calculation, they're using an anecdote as if it were a formula. How to fix it: Saturday's bounce illustrates that noise can reach that size, not that it always does, nor that it's the correct threshold for deciding significance — the correct threshold depends on the sample size and gets calculated with lessons 3 and 6's z-test, not estimated by eye from a single extreme day.

Exercises

Exercise 1 — Find the day closest to the observed lift. Of control's seven days, which one comes closest, in magnitude, to the +18.75% lift observed between control and variant in module 5? What does that imply for whether that lift is "big" or not?

See solution

Saturday, at +20.3% relative to the weekly average, comes closest — in fact, it exceeds it. That implies that, with just control's natural noise, it's already possible to observe a jump as big as (or bigger than) the full lift between control and variant. This doesn't prove the +18.75% is pure chance —that needs lessons 3 and 6's formal calculation— but it does show "it looks big" isn't, on its own, a sufficient argument: background noise can also look big.

Exercise 2 — Calculate the bounce's range. Looking at the seven days' "vs weekly average" column, what's the full range of variation control had during the week (from the lowest value to the highest)?

See solution

The lowest value is -12.6% (Monday) and the highest is +20.3% (Saturday) — a total range of 32.9 percentage points of relative variation, within the same group, with no change involved. That full range is this lesson's most direct piece of evidence: sampling noise, with ~1,700 visits a day, isn't a marginal effect — it's big enough that any comparison between two groups needs a more rigorous criterion than "it looks different".

Exercise 3 — Predict with judgment. If Mercado had measured control with 170,000 visits a day instead of ~1,700 (a hundred times more sample), would you expect the day-to-day variation range to be bigger, smaller, or the same? Justify in one sentence, without calculating the exact formula yet (that comes in module 7).

See solution

Smaller — much smaller. The reason, previewed in this lesson's "Why a small sample bounces more" section: with more people in the calculation, individual decisions to buy or not buy get averaged over many more people, and a few users' fluctuations weigh proportionally less on the final rate. Module 7 is going to formalize this intuition with an exact formula —a proportion's standard error shrinks with the square root of the sample size— but the direction of the effect (more sample, less relative noise) can already be predicted with this lesson's reasoning alone.

Summary and next step

In this lesson you saw, with real numbers from Mercado's control week, that sampling noise alone produces variation as big as (or bigger than) the +18.75% lift observed between control and variant: on Saturday, with no experiment running, conversion already jumped +20.3% relative to the weekly average. You understood, in general terms, why a small sample bounces more than a large one, and why "it looks big" isn't enough as a criterion for telling a real effect apart from sampling coincidence.

Before moving on you should be able to: explain, in your own words, why Mercado's control group bounces day to day with nobody changing anything; and anticipate, even intuitively, that a more rigorous criterion than "the lift looks big" is needed to decide whether recommendations truly works.

Lesson 3 turns that intuition into formal math: the null hypothesis (the precise version of "control and variant come from the same true rate") and the p-value (the exact probability of seeing a lift this big, or bigger, if that hypothesis were true) — calculated, for the first time in the module, over Mercado's own numbers.

Resources

• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments — experimentguide.com. Its introductory chapter on statistical variation explains, with more industry examples, why sampling variance is unavoidable in any online experiment. In English.
• Wikipedia, "Sampling error" — en.wikipedia.org/wiki/Sampling_error. The formal definition of the phenomenon this lesson showed with Mercado's data: the difference between a statistic calculated over a sample and the population's real value. In English.
• Evan Miller, "How Not To Run An A/B Test" — evanmiller.org/how-not-to-run-an-ab-test.html. Revisits this same idea —how much a result bounces from pure chance— to explain why looking at an experiment too often multiplies false alarms; module 7 goes deeper into this. In English.