Statistical significance vs. practical importance

Overview

abTest() is already built and verified (lesson 6): given a control and a variant, it returns a trustworthy significant: true or false verdict. It might seem like that's where this module's work ends — but this lesson shows, with a deliberately constructed hypothetical case, that "significant" isn't a synonym for "important". You're going to run the same function, without changing a single line, over a scenario where the verdict is significant: true and yet no reasonable product team should get excited about the result.

How this connects to the module. This is lesson 4's mistake 3, revisited now with concrete numbers instead of just prose. It's also the last conceptual piece lesson 8's project needs: when reporting recommendations's result, citing significant: true isn't enough — you also need to argue that the effect's size (0.135pp to 1.065pp, a +18.75% relative) is big enough to justify the decision to ship.

An analogy: the pen that weighs a thousandth of a gram more

Picture a lab scale so precise it can tell the weight of two nearly identical pens apart with absolute statistical confidence: pen A weighs, on average, 0.001 grams more than pen B, measured over thousands of repetitions, with a confidence interval that doesn't cross zero anywhere. The result is, in the strictest sense, statistically significant — there's solid, repeatable evidence the difference is real, not measurement noise.

And yet, the obvious question is: who cares? No person can feel a 0.001-gram difference holding a pen in their hand. No manufacturing process needs to distinguish that level of precision. The difference is real —the scale isn't lying— but it's irrelevant to any decision anyone is going to make with that information. That's exactly the problem this lesson solves: statistical significance answers "is the difference real, or is it noise?"; practical importance answers a completely different question: "is the difference big enough for doing something about it to be worth it?". abTest() only answers the first.

Worked example: the same function, a case where "significant" isn't enough

abTest() doesn't change between this lesson and the previous one — it's exactly the same function from lesson 6, copied without modifying a single line. What changes is the case fed into it: instead of Mercado's 12,000 users per variant, imagine a much bigger company, with 5 million users per variant, testing a minor change —say, a copy tweak on a secondary button— that produces a lift of barely one relative percentage point, from 3.01% to 3.04% conversion:

// The SAME abTest() from lesson 6, unchanged, applied to a hypothetical case -- a
// company with 5 million users per variant and a lift of just 0.03 percentage
// points -- to contrast STATISTICAL significance with PRACTICAL importance.
function erf(x) {
  const sign = x < 0 ? -1 : 1;
  x = Math.abs(x);
  const a1 = 0.254829592, a2 = -0.284496736, a3 = 1.421413741,
        a4 = -1.453152027, a5 = 1.061405429, p = 0.3275911;
  const t = 1 / (1 + p * x);
  const y = 1 - (((((a5 * t + a4) * t) + a3) * t + a2) * t + a1) * t * Math.exp(-x * x);
  return sign * y;
}
function normalCdf(x) {
  return 0.5 * (1 + erf(x / Math.sqrt(2)));
}
function abTest({ control, variant }) {
  const { n: n1, conv: conv1 } = control;
  const { n: n2, conv: conv2 } = variant;
  const p1 = conv1 / n1;
  const p2 = conv2 / n2;
  const lift = (p2 - p1) / p1;
  const pooled = (conv1 + conv2) / (n1 + n2);
  const sePooled = Math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2));
  const z = (p2 - p1) / sePooled;
  const pValue = 2 * (1 - normalCdf(Math.abs(z)));
  const seDiff = Math.sqrt((p1 * (1 - p1)) / n1 + (p2 * (1 - p2)) / n2);
  const diff = p2 - p1;
  const ci = [diff - 1.96 * seDiff, diff + 1.96 * seDiff];
  return { rateControl: p1, rateVariant: p2, lift, z, pValue, ci, significant: pValue < 0.05 };
}

// Hypothetical case: 5,000,000 users per variant, control 3.01%, variant 3.04%.
const huge = abTest({
  control: { n: 5_000_000, conv: 150_500 },
  variant: { n: 5_000_000, conv: 152_000 },
});

console.log('=== Hypothetical case: giant N, tiny lift ===\n');
console.log('rateControl = ' + (huge.rateControl * 100).toFixed(3) + '%');
console.log('rateVariant = ' + (huge.rateVariant * 100).toFixed(3) + '%');
console.log('lift        = ' + (huge.lift * 100).toFixed(2) + '%  (relative)');
console.log('difference  = ' + ((huge.rateVariant - huge.rateControl) * 100).toFixed(3) + ' percentage points (absolute)');
console.log('z           = ' + huge.z.toFixed(4));
console.log('pValue      = ' + huge.pValue.toFixed(6));
console.log('CI 95%      = [' + (huge.ci[0] * 100).toFixed(4) + 'pp, ' + (huge.ci[1] * 100).toFixed(4) + 'pp]');
console.log('significant = ' + huge.significant);

What to expect. When you run the file with Node, the output is exactly this:

=== Hypothetical case: giant N, tiny lift ===

rateControl = 3.010%
rateVariant = 3.040%
lift        = 1.00%  (relative)
difference  = 0.030 percentage points (absolute)
z           = 2.7695
pValue      = 0.005615
CI 95%      = [0.0088pp, 0.0512pp]
significant = true

There's the apparent contradiction, resolved with real numbers: significant = true, with a p-value (0.0056) even smaller than Mercado's (0.0114), and a CI that doesn't cross zero. By this module's strict definition, it's a result just as solid —or more so— than recommendations's. And yet, the absolute difference is just 0.03 percentage points — thirty times smaller than Mercado's 0.6pp—. With 5 million users per variant, the sample is so enormous that even a tiny difference, the size of the analogy's pen, becomes detectable with total statistical confidence. The question significant: true never answers is whether it's worth anyone's while to act on those 0.03 points: does it justify the engineering cost, cross-team coordination, and risk of breaking something, for such a small effect? No statistical formula answers that question — the business does.

Why the giant N makes this possible

It's no coincidence this example uses 5 million users per variant instead of 12,000. The z-score depends on dividing the observed difference by the standard error, and the standard error shrinks with the square root of n — the bigger the sample, the smaller the standard error, and the easier it becomes for even a tiny difference to cross the significance threshold. Taken to the extreme: with a big enough sample, almost any difference from zero, however microscopic, eventually becomes "statistically significant" — because significance only asks "is it different from zero?", never "is it different from zero by an amount anyone cares about?". This is why the biggest companies on the internet, with hundreds of millions of users, need a criterion beyond the p-value to decide what to ship — and that additional criterion is, precisely, the effect's size and its business relevance, not significance on its own.

How "worth it" gets decided, in practice

There's no universal formula for "practical importance" —unlike statistical significance, which does have a precise mathematical threshold— but there are concrete questions that replace vague intuition with an explicit criterion:

• How much does the effect represent in business units, not just percentage points? A 0.6pp lift over 24,000 users/week is ~144 additional purchases a week —a number you can compare against the feature's build-and-maintenance cost—. A 0.03pp lift over that same base would be just ~7 additional purchases a week: probably not even enough to cover the change's coordination cost.
• Does the confidence interval's lower end stay relevant? Mercado's CI runs from 0.135pp to 1.065pp — even in the most conservative scenario (the low end), the effect remains substantial. The hypothetical case's CI runs from 0.0088pp to 0.0512pp — not even the high end is big.
• Does this effect hold up against the opportunity cost? The engineering time spent building, shipping, and measuring the experiment could have gone into another bet with more potential — the question isn't just "did it work?", but "did it work well enough to have been the best possible bet?".

These questions don't replace the z-test — they complement it. Mercado answers all three well: the effect is significant, translates into a concrete business number, and holds up even in the CI's conservative scenario.

Common mistakes

Using the p-value as a proxy for the effect's size. What happens: someone compares Mercado's result (p = 0.0114) with this lesson's hypothetical case (p = 0.0056) and concludes the second "is a stronger effect" because its p-value is smaller. Why it happens: it's tempting to rank results by p-value, as if it were a scale of "how important the finding is" — it's the same mistake lesson 3 already warned about, now with a concrete example that makes it obvious. How to spot it: if two experiments get compared by their p-value without looking at the effect's size (the absolute lift, the CI), the comparison is poorly grounded. How to fix it: the p-value depends on both the effect's size and the sample's size — a small effect with a giant sample can give a smaller p-value than a big effect with a modest sample, exactly as in this lesson's example. To compare how "important" a finding is, look at the lift and the CI, not the p-value.

Rejecting any result with a small lift, without calculating the real business impact. What happens: the team sees a 1% relative lift (like the hypothetical case) and dismisses it outright as "insignificant", without translating it into concrete business units. Why it happens: a relative 1% sounds small in a vacuum, but "small" or "big" depends entirely on the business's scale — 1% of a base of millions of users can still be a relevant number in some contexts. How to spot it: if the conversation uses the word "small" or "insignificant" without having calculated how many conversions, how much GMV, or how much revenue that percentage represents over the real traffic base, the judgment is more intuition than analysis. How to fix it: always translate the lift into a concrete business unit before dismissing or celebrating it —as this lesson did with "additional purchases per week"— and compare that number against the real cost of building and maintaining the change.

Exercises

Exercise 1 — Compare the two cases side by side. Complete the table with Mercado's data (lesson 6) and this lesson's hypothetical case: pValue, absolute difference in percentage points, and whether the result, in your opinion, is worth shipping. Justify your decision in a couple of sentences.

See solution

	Mercado	Hypothetical case
pValue	0.0114	0.005615
Absolute difference	0.600pp	0.030pp
Significant?	Yes	Yes
Worth shipping?	Yes	Probably not

Mercado is worth it because, besides being significant, the effect translates into a concrete, substantial business number (~144 additional purchases a week, over the experiment's traffic base) and the CI, even at its most conservative end, remains relevant. The hypothetical case, though just as significant (in fact with a smaller p-value), represents a difference twenty times smaller in absolute terms — the kind of effect that probably doesn't justify the cost of coordinating a launch, unless the company operates at a scale where even hundredths of a point represent relevant sums.

Exercise 2 — Find the breaking point. Without running Node yet, what would you expect to happen to the hypothetical case's pValue if the sample were 50,000 users per variant instead of 5,000,000 (a hundred times smaller), keeping the same rates (3.01% and 3.04%)? Verify by running the code with that change.

See solution

The pValue should grow considerably, probably well above 0.05 — with a sample a hundred times smaller, the standard error grows (roughly by a factor of √100 = 10), which shrinks the z-score by that same order of magnitude, and a much smaller z-score corresponds to a much bigger p-value. Running the code with n: 50_000 and the same rates, the result gives significant: false — the same 1% relative difference, with less sample, stops being distinguishable from noise. This confirms the "Why the giant N makes this possible" section's explanation: this case's significance depended almost entirely on the sample's enormous size, not on the effect being particularly strong.

Exercise 3 — Design the opposite argument. Describe, in a couple of sentences, a scenario where a small relative lift (under 2%) DOES justify shipping, despite sounding unimpressive at first glance. Hint: think about the absolute size of the user base it applies to.

See solution

There's no single correct answer. A reasonable example: a payments platform processing $10 billion a year in transactions detects, with a well-designed experiment, that a checkout flow change produces a lift of just 0.5% in payment conversion rate. Over such a large base, that relative 0.5% translates into tens of millions of additional dollars in processed volume per year — a huge business impact, even though the percentage itself sounds modest. This lesson's central point isn't that "small lifts never matter" — it's that the relative percentage, on its own, doesn't say whether something matters: it needs to be translated to the business's real scale, as the previous section asks.

Summary and next step

In this lesson you ran abTest() —unchanged— over a hypothetical case with a giant N and a tiny lift, and confirmed a result can be significant: true, with a p-value even smaller than Mercado's, and still represent an effect so small (0.03pp) it probably doesn't justify acting on. You learned statistical significance answers "is it real?" while practical importance answers "is it worth it?" — two different questions that each require their own criterion: the p-value and the CI for the first, translation into concrete business units for the second.

Before moving on you should be able to: explain, with the pen analogy, why a result can be significant and trivial at the same time; calculate a given lift's business impact, translating it into a concrete unit; and argue, with your own judgment, why Mercado's lift is worth it while this lesson's hypothetical case probably isn't.

With the module's six content lessons complete —sampling noise, the null hypothesis and the p-value, its most common misunderstandings, the confidence interval, the verified z-test, and now the difference between significant and important— lesson 8 brings it all together in the final project: the recommendations launch's full significance report, with the decision to ship, revert, or iterate.

Resources

• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments — experimentguide.com. The book dedicates a full chapter to the distinction between statistical significance and practical relevance in the context of large-scale industry experimentation. In English.
• Ronald L. Wasserstein and Nicole A. Lazar, "The ASA Statement on p-Values: Context, Process, and Purpose" — tandfonline.com/doi/full/10.1080/00031305.2016.1154108. One of its central principles explicitly warns against confusing statistical significance with an effect's scientific or practical importance. In English.
• Optimizely, "Statistical significance" — support.optimizely.com/hc/en-us/articles/4410284003341-Statistical-significance. Explains, from the perspective of an experimentation platform used by massive-scale companies, why sample size can make almost any effect "significant". In English.