The null hypothesis and the p-value

Overview

The previous lesson left a clear but informal intuition: sampling noise, on its own, can already produce swings the size of the lift observed between control and variant. This lesson turns that intuition into two precise, calculable tools anyone can reproduce with the same data: the null hypothesis (nullHypothesis), the formal version of "there's no real effect, control and variant come from the same true rate"; and the p-value (pValue), the number that measures how surprising the observed result would be if that hypothesis were true. By the end of this lesson you're going to have, for the first time in the module, a z-score and a p-value calculated over the recommendations A/B test's real data.

How this connects to the module. This lesson builds the mathematical core the rest of the module is going to refine and package: lesson 4 corrects the most common misunderstandings about the number you're about to calculate here; lesson 5 adds the confidence interval as a complement; and lesson 6 takes this exact same logic —null hypothesis, pooled proportion, z-score, p-value— and packages it into abTest(), the module's verified, reusable function.

An analogy: the jury that presumes innocence

In a trial, the jury doesn't start by asking "is this person guilty?". It starts by assuming, by default, that the defendant is innocent — that's the starting hypothesis, and it holds until the presented evidence is strong enough to rule it out "beyond reasonable doubt". The prosecutor doesn't have to prove innocence; they have to accumulate evidence so improbable under the assumption of innocence that it becomes reasonable to abandon that presumption.

The null hypothesis works exactly the same way, under a different name. recommendations has no real effect — that's the starting presumption, the same one from lesson 1's coin analogy: "the coin is fair until proven otherwise". The experiment doesn't have to prove there's an effect; it has to accumulate a difference so improbable under the assumption of "no effect" that it becomes reasonable to abandon that presumption. The p-value is, in this analogy, the equivalent of "how convincing is the prosecutor's evidence, measured on a scale of 0 to 1" — the smaller it is, the more uncomfortable the evidence is for the innocence hypothesis (or, in Mercado's case, for the "no effect" hypothesis).

And there's an important asymmetry, the same one that exists in a courtroom: not finding enough evidence to convict isn't the same as proving innocence. A jury that says "not guilty" isn't saying "definitely didn't do it" — it's saying "the evidence presented didn't clear the threshold". Hold onto this idea; lesson 4 is going to come back to it in full detail, because it's one of experimentation's most expensive mistakes.

Worked example: the z-score and p-value of Mercado's A/B test

The recommendations experiment's null hypothesis is written like this: nullHypothesis: control's and variant's true conversion rate is the SAME. Under that hypothesis, the best estimate of that shared rate isn't rateControl or rateVariant separately — it's the two samples' combined proportion, the pooled proportion: if control and variant are, underneath, the same thing, the most honest way to estimate that shared rate is to pool all the conversions and all the visits from both groups into a single calculation.

With that pooled rate you can calculate the expected standard error of the difference between two samples of those sizes —how much rateControl and rateVariant should bounce relative to each other, just by chance, if they truly came from the same rate—. Dividing the observed difference by that standard error gives the z-score: how many "standard errors of expected noise" of distance there are between what was observed and what the null hypothesis predicts (a difference of zero).

// First calculation of the z-score and p-value over Mercado's A/B (control vs variant).
// Not yet packaged into abTest(), and no confidence interval yet -- that comes in L5/L6.
// Phi (standard normal CDF) via the Abramowitz & Stegun 7.1.26 approximation of the
// error function: Phi(x) = 0.5 * (1 + erf(x / sqrt(2))).
function erf(x) {
  const sign = x < 0 ? -1 : 1;
  x = Math.abs(x);
  const a1 = 0.254829592, a2 = -0.284496736, a3 = 1.421413741,
        a4 = -1.453152027, a5 = 1.061405429, p = 0.3275911;
  const t = 1 / (1 + p * x);
  const y = 1 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-x * x);
  return sign * y;
}
function normalCdf(x) {
  return 0.5 * (1 + erf(x / Math.sqrt(2)));
}

// Module 5's data: control 384/12000 (3.2%), variant 456/12000 (3.8%), lift +18.75%
const control = { n: 12000, conv: 384 };
const variant = { n: 12000, conv: 456 };

const p1 = control.conv / control.n;
const p2 = variant.conv / variant.n;

// H0: there's no real difference -- p1 and p2 come from the SAME true rate. Under H0,
// the best estimate of that shared rate is the pooled proportion.
const pooled = (control.conv + variant.conv) / (control.n + variant.n);
const sePooled = Math.sqrt(pooled * (1 - pooled) * (1 / control.n + 1 / variant.n));
const z = (p2 - p1) / sePooled;

// TWO-tailed p-value: the probability of seeing a |z| this big or bigger, if H0 were true.
const pValue = 2 * (1 - normalCdf(Math.abs(z)));

console.log('=== z-score and p-value of Mercado\'s A/B (control vs variant) ===\n');
console.log('rateControl = ' + control.conv + '/' + control.n + ' = ' + (p1 * 100).toFixed(2) + '%');
console.log('rateVariant = ' + variant.conv + '/' + variant.n + ' = ' + (p2 * 100).toFixed(2) + '%');
console.log('pooled      = ' + (pooled * 100).toFixed(4) + '%   (shared rate assumed under H0)');
console.log('sePooled    = ' + sePooled.toFixed(6));
console.log('z           = ' + z.toFixed(4));
console.log('pValue      = ' + pValue.toFixed(4) + '  (two-tailed)');
console.log('\np < 0.05? ' + (pValue < 0.05 ? 'yes' : 'no'));

What to expect. When you run the file with Node, the output is exactly this:

=== z-score and p-value of Mercado's A/B (control vs variant) ===

rateControl = 384/12000 = 3.20%
rateVariant = 456/12000 = 3.80%
pooled      = 3.5000%   (shared rate assumed under H0)
sePooled    = 0.002373
z           = 2.5289
pValue      = 0.0114  (two-tailed)

p < 0.05? yes

Read the result carefully, word by word, because the exact phrasing matters: if recommendations had no real effect at all (the null hypothesis), the probability of observing a conversion difference as big, or bigger, than the one seen between control and variant is only 1.14%. It isn't impossible under the null hypothesis —it never is— but it's uncommon: it would happen, by pure chance, roughly 1 out of every 88 experiments if there truly were no effect at all. With the conventional alpha = 0.05 (5%) threshold, 1.14% falls below it — the result is statistically significant: there's enough evidence to doubt the null hypothesis.

Why "two-tailed"

The calculation used 2 * (1 - Phi(|z|)) instead of just 1 - Phi(z). The reason: before running the experiment, Mercado's team didn't know whether recommendations was going to raise or lower conversion —it was a reasonable bet, but not a certainty—. A two-tailed test accounts for both possible directions: it asks "how surprising is a difference of this size, in either direction?", not just "how surprising is a rise of this size?". That's the correct approach when the original question was honestly open in both directions, which is the normal case for a product A/B test — nobody launches an experiment 100% certain, in advance, that the result can only go one way.

Common mistakes

Calculating the z-score with a single sample's standard error, not the pooled one. What happens: someone calculates rateControl's standard error alone, or rateVariant's alone, and uses it for the z-score, instead of combining both samples under the null hypothesis. Why it happens: it seems simpler to use "the" standard error of a rate, without thinking that the null hypothesis specifically assumes both rates are the same, and that assumption changes how the correct standard error gets calculated. How to spot it: if the standard error calculation doesn't use pooled (the combined proportion of conversions and visits from both groups), the resulting z-score doesn't correspond to the standard null-hypothesis test. How to fix it: for the z-score and p-value, always use the pooled proportion — it's the standard formula for a two-proportions z-test, and it's the one abTest() uses in lesson 6. (Lesson 5's confidence interval, on the other hand, does use the non-pooled standard error — for a different reason, which that lesson explains.)

Thinking the p-value measures the effect's size. What happens: someone compares two experiments, sees one has p=0.01 and the other p=0.04, and concludes the first has "a bigger" or "more important" effect than the second. Why it happens: it's tempting to treat the p-value as a scale of "how strong the effect is", as if it were a thermometer. How to spot it: if someone says "this is more significant, so the business impact is bigger", they're mixing up two different questions. How to fix it: the p-value measures confidence that some effect exists, not that effect's size — two experiments with the same real lift can have very different p-values just from differences in sample size, and two experiments with similar p-values can have lifts of completely different sizes. The effect's size is measured by the lift (module 5) and, above all, the confidence interval (this module's lesson 5) — not the p-value on its own. This mistake gets revisited in more depth in lesson 4.

Exercises

Exercise 1 — Calculate the pooled proportion by hand. With control = {n: 12000, conv: 384} and variant = {n: 12000, conv: 456}, calculate pooled's value by hand before running the code. Verify your result against the lesson's output.

See solution

pooled = (384 + 456) / (12000 + 12000) = 840 / 24000 = 0.035 = 3.5%. It matches exactly the output's pooled = 3.5000% — that makes sense, because 3.5% sits right between control's 3.2% and variant's 3.8% (and, since both groups are the same size, pooled is simply the average of the two rates).

Exercise 2 — Interpret the p-value in your own words. Without copying the lesson's sentence, write in one sentence what pValue = 0.0114 means for Mercado's experiment. Avoid the words "probability that the null hypothesis is true" — that phrase, even though it sounds similar, is incorrect (lesson 4 explains why).

See solution

A correct phrasing: "If recommendations had no real effect on conversion, seeing a difference as big —or bigger— than the one observed between control and variant would happen, from pure sampling chance, only 1.14% of the time." The sentence describes the probability of the observed data (or something more extreme) under the null hypothesis, not the probability that the null hypothesis is true or false — that exact distinction is the next lesson's entire topic.

Exercise 3 — Predict with judgment. If Mercado's experiment had run with only 1,200 users per variant (a tenth of the size), instead of 12,000, keeping the same conversion rates (3.2% and 3.8%), would you expect the z-score to be bigger, smaller, or the same? And the p-value?

See solution

The z-score would be smaller (closer to 0), and the p-value would be bigger (less significant). The reason: sePooled depends on 1/n1 + 1/n2 — with samples ten times smaller, that term grows, the standard error grows, and dividing the same difference (p2 - p1) by a bigger standard error gives a smaller z-score. A smaller z-score sits closer to the normal distribution's center, so normalCdf gets closer to 0.5 and the p-value (2 * (1 - normalCdf(|z|))) grows. This is lesson 2's same idea —less sample, more relative noise— now expressed in the z-test's exact formula; module 7 revisits it to calculate, in reverse, how much sample is needed for a given lift.

Summary and next step

In this lesson you calculated, for the first time in the module, the z-score (2.5289) and the p-value (0.0114) of the recommendations real A/B test, using the formal null hypothesis —"control and variant come from the same true rate"— and the pooled proportion to estimate the standard error under that hypothesis. With alpha = 0.05 as the conventional threshold, the result is statistically significant: 0.0114 < 0.05.

Before moving on you should be able to: explain, with the jury analogy, what the null hypothesis represents; calculate the pooled proportion by hand given two groups; and precisely state what a p-value of 0.0114 means, without falling into the incorrect phrase "probability that the null hypothesis is true".

That incorrect phrase —and two others just as common and just as expensive— are exactly lesson 4's topic: what the p-value does NOT mean, before you keep building on it.

Resources

• Wikipedia, "p-value" — en.wikipedia.org/wiki/P-value. Defines the p-value exactly the way this lesson used it: "the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct". In English.
• Wikipedia, "Z-test" — en.wikipedia.org/wiki/Z-test, the "Comparing the proportions of two binomials" section. The exact two-proportions z-test formula this lesson's code implements, with the same pooled-standard-error structure. In English.
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments — experimentguide.com. The chapter on hypothesis testing develops, in more depth, the "assume H0, measure how surprising the data is" logic this lesson presented with the jury analogy. In English.