The confidence interval

Overview

Lesson 4 ended by pointing out a gap: the p-value says whether there's evidence of an effect, but it doesn't say how much that effect is worth or within what range of uncertainty. This lesson fills that gap with the 95% confidence interval (confidenceInterval): instead of a single "yes or no" number, a full range where the real conversion difference between control and variant probably lives. By the end of this lesson you'll be able to read a CI and decide significance with a rule as simple as checking whether the range crosses zero.

How this connects to the module. Lesson 3's z-score and p-value and this lesson's confidence interval are two halves of the same calculation, done with an important technical difference this lesson explains. Lesson 6 is going to bring both halves together —z-score, p-value, and CI— into a single verified abTest() function; this lesson is the intermediate step that makes sure you understand each piece before they get combined.

An analogy: tomorrow's weather, not just "yes or no is it going to rain"

A weather forecast that says "yes, it's going to rain tomorrow" is useful, but incomplete. A forecast that says "it's going to rain between 5 and 20 millimeters, with 95% confidence" is much more useful — it tells you not just that something's going to happen, but how much, and how sure the meteorologist is about that range. If the range were "between -3 and 20 millimeters" (a range including negative values, which make no sense for rain, but serve as an example), the honest conclusion would be "we're not sure whether it's going to rain at all" — the range crosses the point where "nothing happens" (zero millimeters).

The A/B test's confidence interval does exactly the same thing with the conversion difference between control and variant. Instead of just "yes, it's significant" or "no, it isn't", it gives you the full range of plausible values for that real difference, with 95% confidence. If that range includes zero —"the real difference could be zero, or even negative"—, there's no solid evidence of an effect. If the entire range sits above zero —"the real difference, almost certainly, is positive, somewhere between these two values"—, then there is evidence of a real effect, and you also know roughly how much it's worth.

Worked example: the 95% CI of the difference at Mercado

The confidence interval gets built around the observed difference (p2 - p1), adding and subtracting a margin: 1.96 standard deviations of that difference's error —1.96 is the constant corresponding to 95% confidence under the normal distribution, the same one you already saw as Phi(1.96) ≈ 0.975 in the previous lesson—. The key difference from the z-score calculation is the standard error used: here pooled does not get used, because the CI isn't asking "how unusual would this be if the two rates were equal?" (that assumes the null hypothesis) — it's asking "what's the plausible range for the real difference, whatever it is?", a question that doesn't need to assume both rates are equal, so each sample contributes its own variance separately:

// 95% confidence interval of the DIFFERENCE (p2 - p1), over Mercado's A/B.
// Unlike the z-score/p-value (which use the pooled SE, assuming H0), the CI uses the
// NON-pooled SE: each rate contributes its own variance, because the CI doesn't assume
// p1 and p2 are equal -- on the contrary, it describes the plausible range of their real
// DIFFERENCE.
const control = { n: 12000, conv: 384 };
const variant = { n: 12000, conv: 456 };

const p1 = control.conv / control.n;
const p2 = variant.conv / variant.n;
const diff = p2 - p1;

const seDiff = Math.sqrt((p1 * (1 - p1)) / control.n + (p2 * (1 - p2)) / variant.n);
const ci95 = [diff - 1.96 * seDiff, diff + 1.96 * seDiff];

console.log('=== 95% confidence interval of the difference (Mercado) ===\n');
console.log('diff (p2 - p1) = ' + (diff * 100).toFixed(3) + ' percentage points');
console.log('seDiff         = ' + seDiff.toFixed(6));
console.log('CI 95%         = [' + (ci95[0] * 100).toFixed(3) + 'pp, ' + (ci95[1] * 100).toFixed(3) + 'pp]');
console.log('\nDoes it cross 0? ' + (ci95[0] <= 0 && ci95[1] >= 0 ? 'yes (would not be significant)' : 'no (consistent with significant)'));

What to expect. When you run the file with Node, the output is exactly this:

=== 95% confidence interval of the difference (Mercado) ===

diff (p2 - p1) = 0.600 percentage points
seDiff         = 0.002372
CI 95%         = [0.135pp, 1.065pp]

Does it cross 0? no (consistent with significant)

Read it like this: with 95% confidence, the real conversion difference between having recommendations and not having them is between 0.135 and 1.065 percentage points — always in variant's favor. The entire range is positive: not even the lowest end (0.135pp) reaches zero, let alone crosses it. That's exactly what lesson 3 had already anticipated with the p-value: there's evidence of a real effect, and now, on top of that, you know that effect's plausible size ranges from "modest" (0.135pp) to "considerable" (1.065pp) — a range nearly ten times wider at its upper end than at its lower one, which is honest: with this sample, the effect's exact size can't be pinned down any further, only its plausible range.

Why the CI and the p-value almost always agree (and what to do when they don't)

It's no coincidence that Mercado's CI doesn't cross zero right when the p-value had already come out significant (p = 0.0114 < 0.05) — the two calculations are deeply connected: a 95% confidence interval that doesn't cross zero is, in practice, mathematically equivalent to a two-tailed p-value below 0.05. You can think of the CI as a more informative way of asking the same question: instead of "does the p-value cross the 0.05 threshold?", "does the interval cross the 'no effect' value (zero)?". Both questions, almost always, give the same yes-or-no answer — the CI's advantage is that, besides answering that question, it also tells you how much.

The small technical difference between pooled (for z/p) and non-pooled (for the CI) almost never changes the significance conclusion in practice —with samples the size of Mercado's, both standard-error approaches give very similar results— but it's worth remembering: if you ever see a CI and a p-value that seem to contradict each other right at the threshold's edge (p very close to 0.05, CI with one end very close to zero), that's a sign the result sits exactly on the boundary, not that something was miscalculated.

Common mistakes

Reading only whether the CI crosses zero, ignoring the range's width. What happens: someone looks at Mercado's CI, sees it doesn't cross zero, says "significant" and moves on without noticing the range goes from 0.135pp to 1.065pp — a fairly wide range for this sample's size. Why it happens: the yes/no question gets resolved quickly and feels like the complete answer, when the interval's width is information just as valuable. How to spot it: if nobody mentions the CI's width when reporting the result —only "it's significant" or "it isn't"—, half the information the interval offers is being wasted. How to fix it: always report the full interval, not just the binary conclusion — a narrow CI (0.5pp to 0.7pp) inspires much more confidence in the effect's exact size than a wide one like Mercado's (0.135pp to 1.065pp), even though both are equally "significant".

Using the pooled standard error to calculate the confidence interval. What happens: someone reuses lesson 3's sePooled (calculated under the null hypothesis) to build the CI, instead of this lesson's non-pooled seDiff. Why it happens: there's already a standard-error variable calculated earlier in the code, and reusing it seems simpler than calculating a new one. How to spot it: if the CI was calculated with the same standard-error variable as the z-score, sePooled was probably used by mistake — check that the CI's formula uses p1*(1-p1)/n1 + p2*(1-p2)/n2 (each sample separately), not the pooled version. How to fix it: remember the underlying reason explained in this lesson — the z-score/p-value assume H0 (which is why they use a common, pooled rate) while the CI describes the difference's plausible range without assuming it's zero (which is why each sample contributes its own variance). They're two different questions, and each needs its own standard error.

Exercises

Exercise 1 — Verify "doesn't cross zero" by hand. With ci95 = [0.00135, 0.01065] (as a fraction, not in percentage points), explain in one sentence why this interval "doesn't cross zero" and what would need to be different for it to cross it.

See solution

The interval doesn't cross zero because its lower end, 0.00135, is already positive — the entire range, from the low end to the high end, sits above zero. For the interval to cross zero, the lower end would have to be negative (for example, [-0.001, 0.01]), which would happen if the observed difference (diff) were smaller, or if the standard error (seDiff) were bigger —with a smaller sample, for example— so that subtracting 1.96 * seDiff from diff crossed below zero.

Exercise 2 — Calculate the CI of a hypothetical case. With control = {n: 12000, conv: 390} and variant = {n: 12000, conv: 396} (a much smaller difference than Mercado's real one), would you expect the difference's CI to cross zero or not? Verify by running this lesson's code with these numbers.

See solution

With such a small difference (diff = 6/12000 = 0.05pp) and the same sample size as Mercado's, it's reasonable to expect the CI to cross zero — the margin of error (1.96 * seDiff) with these sample sizes is around 0.46pp, much bigger than the observed difference of just 0.05pp. Running the code with these values, the resulting CI is approximately [-0.42pp, 0.52pp] — it clearly crosses zero, consistent with a non-significant result. This illustrates the same point as lesson 2: a small difference, with this sample, can't be told apart from noise.

Exercise 3 — Predict with judgment. If Mercado had run the same experiment with double the sample (24,000 per variant) keeping the same observed rates (3.2% and 3.8%), would you expect the CI's width to be narrower, wider, or the same? Justify without calculating the exact number.

See solution

Narrower. The CI's width depends directly on seDiff, which in turn depends on 1/n1 and 1/n2 inside the square root — with double the sample, those terms halve, and seDiff shrinks by a factor of roughly 1/√2 (not exactly by half, because it's inside a square root). A smaller seDiff means a smaller margin (1.96 * seDiff), and therefore a narrower interval around the same observed difference. This is the same relationship between sample and precision you're going to formalize with sampleSize's exact formula in module 7.

Summary and next step

In this lesson you calculated the 95% confidence interval of the conversion difference in Mercado's A/B: [0.135pp, 1.065pp], a fully positive range that doesn't cross zero — consistent with lesson 3's significant p-value. You learned the technical difference between the pooled standard error (for the z-score and p-value, which assume the null hypothesis) and the non-pooled one (for the CI, which describes the real difference's plausible range without that assumption), and why "the CI doesn't cross zero" is, in practice, the same conclusion as "p < 0.05", just with more information about the effect's size.

Before moving on you should be able to: explain with the weather-forecast analogy what a confidence interval adds over a simple "yes or no"; calculate by hand whether a given CI crosses zero or not; and anticipate how a CI's width would change if the sample were bigger or smaller.

With the z-score, the p-value, and the confidence interval now understood separately, lesson 6 brings them together into a single verified function: abTest(), the full proportions z-test, tested against known cases before trusting it over Mercado's real data.

Resources

• Wikipedia, "Confidence interval" — en.wikipedia.org/wiki/Confidence_interval. The confidence interval's formal definition and its correct frequentist interpretation —an important nuance, similar in spirit to lesson 4's about the p-value—. In English.
• Evan Miller, "Evan's Awesome A/B Tools" — evanmiller.org/ab-testing. Its Chi-Squared Test calculator reports, alongside the significance result, the difference's confidence interval — the same piece this lesson calculated, in a ready-to-use tool. In English.
• Optimizely, "Statistical significance" — support.optimizely.com/hc/en-us/articles/4410284003341-Statistical-significance. Explains, from a real experimentation platform's perspective, how a confidence interval gets communicated alongside the significance result. In English.