Simpson's paradox

Overview

Lessons 4 and 5 showed two ways of fooling yourself through an excess of opportunities —looking too many times, testing too many metrics—. This lesson presents a completely different pitfall, and in a way a more unsettling one: an aggregate result, calculated over a single primary metric, measured once, at the experiment's end, with the correct sample size — and still misleading, because it hides a different story when you split it by segments. It's called Simpson's paradox, and it happens when the trend you see in the total reverses, or outright disappears, when you look at the data split by group.

The name is intimidating, but the mechanism is precise: it happens when a third variable —a confounder— is correlated at the same time with (a) how well each segment converts on its own, and (b) what proportion of each experimental group (control or variant) comes from each segment. If those two correlations line up in a certain way, each group's aggregate average can end up telling a completely different story —sometimes the opposite one— from what happens within each individual segment.

How this connects to the module. This lesson builds simpsonCheck(), the function you're going to reuse in lesson 8's mini-project to audit whether recommendations's +18.75% lift —the same one from modules 5 and 6— holds up when you split traffic by device type (mobile / desktop). Today you work with an illustrative case, deliberately built to show the reversal with total clarity; the mini-project applies the same function to the experiment's real data and checks whether this specific danger happened there.

An analogy: the drug that works… and "fails"

Imagine a clinical trial where a new drug is tested against a placebo. When the team looks only at male patients, the drug clearly wins: it improves more than the placebo. When it looks only at female patients, the drug also clearly wins. Two subgroups, two consistent wins for the drug. And yet, when the team combines all patients into a single aggregate number —men and women mixed—, the placebo appears to win. How is it possible for something to lose in every subgroup separately, but win in the total?

The answer is in how the groups got mixed, not in the drug. If, for example, most of the trial's men (where the drug works very well, but the underlying disease is milder for everyone, men or women) ended up in the placebo group, and most of the women (where the underlying disease is generally more severe, drug or placebo) ended up in the treatment group, the "treatment" aggregate ends up dominated by the harder cases, and the "placebo" aggregate ends up dominated by the milder cases — not because the placebo works better, but because each group's patient mix isn't comparable. The drug is honestly still better in every homogeneous subgroup; it's the aggregate, contaminated by an uneven mix, that lies.

An A/B test can suffer exactly the same phenomenon if the mix of user types —for example, mobile versus desktop— ends up different between control and variant, whether from an instrumentation error, an assignment bug, or simply a sampling coincidence. The aggregate lift you calculated with abTest() in module 5 could be telling a different story than the one each segment tells separately — and the only way to know is, precisely, checking the segments.

Worked example: simpsonCheck() over an illustrative Mercado case

We build a pedagogical case, deliberately designed to show the reversal clearly — not the recommendations experiment's real data (you check that in lesson 8's mini-project). Imagine that, due to an error in how traffic got split during the rollout, variant ended up receiving much more mobile traffic (a channel that generally converts less) than control, which was left with much more desktop traffic (a channel that converts more):

// simpsonCheck: compares the aggregate lift (all traffic together) against the
// per-segment lift, and detects whether there's a reversal (Simpson's paradox) --
// the case where variant wins in EVERY individual segment but loses in the
// aggregate (or vice versa), because of how each segment's traffic got mixed
// between control and variant.
function simpsonCheck(segments) {
  const totals = { control: { n: 0, conversions: 0 }, variant: { n: 0, conversions: 0 } };
  const bySegment = {};

  for (const [name, seg] of Object.entries(segments)) {
    const controlRate = seg.control.conversions / seg.control.n;
    const variantRate = seg.variant.conversions / seg.variant.n;
    bySegment[name] = { controlRate, variantRate, variantWins: variantRate > controlRate };
    totals.control.n += seg.control.n;
    totals.control.conversions += seg.control.conversions;
    totals.variant.n += seg.variant.n;
    totals.variant.conversions += seg.variant.conversions;
  }

  const aggControlRate = totals.control.conversions / totals.control.n;
  const aggVariantRate = totals.variant.conversions / totals.variant.n;
  const aggregateVariantWins = aggVariantRate > aggControlRate;
  const segmentValues = Object.values(bySegment);
  const allSegmentsAgree = segmentValues.every((s) => s.variantWins === segmentValues[0].variantWins);
  const reversal = allSegmentsAgree && segmentValues[0].variantWins !== aggregateVariantWins;

  return {
    bySegment,
    aggregate: { controlRate: aggControlRate, variantRate: aggVariantRate, variantWins: aggregateVariantWins },
    reversal,
  };
}

// ILLUSTRATIVE case (not the real module 5 data): recommendations segmented
// by device, with a confounder -- variant ended up with much more mobile
// traffic (lower-converting channel), control with much more desktop
// (higher-converting channel).
const illustrativeCase = {
  mobile: {
    control: { n: 1000, conversions: 20 },   // 2.00%
    variant: { n: 9000, conversions: 189 },  // 2.10%
  },
  desktop: {
    control: { n: 9000, conversions: 450 },  // 5.00%
    variant: { n: 1000, conversions: 52 },   // 5.20%
  },
};

const result = simpsonCheck(illustrativeCase);
console.log('=== simpsonCheck() -- illustrative case with a device confounder ===\n');
for (const [name, seg] of Object.entries(result.bySegment)) {
  console.log(name + ':  control=' + (seg.controlRate * 100).toFixed(2) + '%  variant=' +
    (seg.variantRate * 100).toFixed(2) + '%  does variant win? ' + seg.variantWins);
}
console.log('\naggregate:  control=' + (result.aggregate.controlRate * 100).toFixed(2) + '%  variant=' +
  (result.aggregate.variantRate * 100).toFixed(2) + '%  does variant win? ' + result.aggregate.variantWins);
console.log('\nreversal detected (Simpson): ' + result.reversal);

What to expect. When you run the file with Node, the output is exactly this:

=== simpsonCheck() -- illustrative case with a device confounder ===

mobile:  control=2.00%  variant=2.10%  does variant win? true
desktop:  control=5.00%  variant=5.20%  does variant win? true

aggregate:  control=4.70%  variant=2.41%  does variant win? false

reversal detected (Simpson): true

Look closely at what happens: in mobile, variant wins (2.10% versus 2.00%). In desktop, variant also wins (5.20% versus 5.00%). The only two segments that exist agree: variant is better. And yet, in the aggregate, control converts at 4.70% and variant at just 2.41% — a complete reversal, variant losing by more than 2 percentage points in the number a careless report would cite as "the experiment's result". What happened? control's aggregate is dominated by desktop (9,000 of its 10,000 users, the higher-converting channel), while variant's aggregate is dominated by mobile (9,000 of its 10,000 users, the lower-converting channel). The aggregate isn't measuring "recommendations's effect" — it's measuring, mostly, the difference between mobile and desktop, disguised as a difference between control and variant.

Why this is a warning sign about randomization, not just the segments

This lesson's most important point isn't "always distrust the aggregate" — it's that a Simpson reversal almost always signals a randomization problem, not a mysterious property of the data. If control and variant were truly randomly assigned —module 5's lesson 4's full discipline—, the mix of mobile and desktop should, on average, be practically identical in both groups: chance has no reason to prefer sending more mobile traffic to one group than the other. Today's illustrative case, with variant at 90% mobile and control at 90% desktop, isn't what a correctly implemented random assignment would produce — it's the kind of imbalance that shows up when something went wrong: a rollout bug, assignment by app version instead of by user, or an instrumentation error that misclassified traffic.

That's why simpsonCheck() serves a double purpose: it doesn't just protect against reporting a misleading aggregate lift, it also acts as an audit of the randomization itself. If you split by a reasonable segment —device, country, user type— and find a reversal like today's, the first question isn't "which segment is the real winner?" — it's "why is this segment's mix so different between control and variant, if the assignment was supposed to be random?". Lesson 8's mini-project applies exactly this audit to the recommendations experiment's real data.

Common mistakes

Trusting the aggregate lift without checking whether the segment mix is balanced between groups. What happens: a team reports checkoutConversionRate's lift calculated with abTest() over the total user base, never checking whether the proportion of mobile/desktop (or any other relevant segment) is similar between control and variant. Why it happens: the aggregate is the simplest number to calculate and communicate, and splitting by segments feels like an optional extra step, not a necessary check. How to spot it: nobody on the team can say, with a concrete number, what percentage of control and variant is mobile versus desktop. How to fix it: as today's simpsonCheck() does, always check the mix of at least one structurally important segment (device, country, user type) before fully trusting the aggregate — it's a quick check that can reveal a serious imbalance, like in today's illustrative case.

Seeing a positive aggregate lift and not bothering to check whether it holds up by segment, "because it already looks good". What happens: recommendations's aggregate lift came out positive and significant (module 6), and the team considers the result closed, without running simpsonCheck() or any segment breakdown, because "the number that matters already came out well". Why it happens: checking the segments of a result that already looks favorable feels like looking for flaws in something that has none, instead of a routine check. How to spot it: the experiment's analysis stops at the aggregate number, with no segment breakdown in the final report. How to fix it: the segment check isn't optional only when the result looks bad — it's part of the full discipline of auditing an experiment, regardless of whether the aggregate favors the hypothesis you were hoping for. Lesson 8's mini-project applies exactly this check to recommendations's real result, regardless of already knowing, from modules 5 and 6, that the aggregate looks favorable.

Upon finding a Simpson reversal, concluding "the effect isn't real" instead of investigating the mix. What happens: a team finds a reversal like today's and incorrectly concludes recommendations "doesn't actually work" — dismissing the per-segment result (where variant does consistently win) in favor of the aggregate (where it appears to lose). Why it happens: the aggregate number feels more "official" or easier to communicate than a segment breakdown, and it's tempting to treat it as the final truth even after finding the reversal. How to spot it: the analysis's conclusion cites the aggregate number as definitive, with no explanation for why it differs so drastically from each individual segment. How to fix it: when you find a reversal, the correct step isn't blindly choosing the aggregate or the segment — it's investigating the reversal's cause (almost always, as you saw today, a mix or randomization problem) and, while that cause remains unresolved, trusting the pattern that consistently repeats in each individual segment more than a potentially mix-contaminated aggregate.

Exercises

Exercise 1 — Detect the reversal with new numbers. A second experiment at Mercado (an "add to cart" button redesign) gives these per-segment results: newUsers — control: n=2,000, conversions=100 (5.0%); variant: n=8,000, conversions=560 (7.0%). returningUsers — control: n=8,000, conversions=640 (8.0%); variant: n=2,000, conversions=220 (11.0%). Calculate control's and variant's aggregate by hand. Is there a Simpson reversal?

See solution

control aggregate: (100+640)/(2000+8000) = 740/10000 = 7.4%. variant aggregate: (560+220)/(8000+2000) = 780/10000 = 7.8%. In this case, variant wins in both segments (newUsers: 7.0% > 5.0%; returningUsers: 11.0% > 8.0%) and in the aggregate (7.8% > 7.4%) — there's no reversal. Running simpsonCheck() with this data would confirm reversal: false. This exercise deliberately contrasts with today's worked example: not every mix difference between segments produces a reversal — it only happens when the mix and each segment's rates line up in a specific way, like in today's device illustrative case.

Exercise 2 — Explain why Exercise 1 had no reversal. Unlike today's worked example (where variant had 90% mobile traffic and control 90% desktop), in Exercise 1 the mix is also different between groups (variant has 80% newUsers, control has 80% returningUsers). Why didn't that mix difference produce a reversal this time?

See solution

For a Simpson reversal to happen, the uneven mix has to combine with an effect direction that "cancels" the pattern when weighted differently. In today's example, control was concentrated in the higher-conversion segment (desktop, 5%) while variant was concentrated in the lower-conversion one (mobile, 2%) — that specific combination is what flips the aggregate. In Exercise 1, even though the mix is also uneven, variant is concentrated in newUsers (the lower of the two segments' rates, 5-7%) and its relative advantage within both segments is big enough that, even weighted by that unfavorable mix, it still wins in the aggregate. The general lesson: an uneven mix between groups is a warning sign worth checking (today's first common mistake), but it doesn't on its own guarantee a reversal — it depends on how each segment's exact rates and weights combine.

Exercise 3 — Design the correct check for the Mercado team. Based on this lesson's three sections, write in 2-3 sentences the check step the recommendations team should add to its checklist, before reporting any aggregate lift as an experiment's final result.

See solution

A reasonable step: "Before reporting any experiment's aggregate lift, run simpsonCheck() (or an equivalent breakdown) over at least one structurally relevant segment —device, country, user type—. If reversal comes out true, or if that segment's mix is noticeably different between control and variant, don't report the aggregate as definitive: first investigate whether there was a randomization problem before deciding which number to trust." This turns the segment check into a mandatory protocol step, exactly like the randomize() balance check module 5 already verified before trusting any result.

Summary and next step

In this lesson you saw Simpson's paradox: an aggregate lift can show a trend —or directly its opposite— compared to what happens in each individual segment, when a confounder (like device type) is imbalanced between control and variant. You built and ran simpsonCheck() over an illustrative case where variant won on mobile (2.10% versus 2.00%) and on desktop (5.20% versus 5.00%) but lost in the aggregate (2.41% versus 4.70%) — a complete reversal caused by an imbalanced traffic mix between the two groups. And you saw why a reversal like that is almost always, underneath, a warning sign about the randomization itself, not a mysterious, inevitable property of the data.

Before moving on you should be able to: explain in your own words how a confounder can reverse an aggregate result; run simpsonCheck() over segmented data and read whether there's a reversal; and explain why a Simpson reversal should first make you suspect the randomization, not dismiss the effect outright.

Lesson 7, the last one before the mini-project, closes the pitfall catalog with two more: the novelty effect —a launch's initial bump that fades over time, easy to confuse with a permanent effect— and p-hacking —the temptation, subtler than it looks, to adjust the analysis after seeing the data until something comes out significant.

Resources

• Wikipedia, "Simpson's paradox" — en.wikipedia.org/wiki/Simpson's_paradox. The formal reference for the phenomenon, with several documented real examples —including the famous case of a kidney stone treatment study, very similar in structure to this lesson's drug example—. In English.
• Brilliant.org, "Simpson's Paradox" — brilliant.org/wiki/simpsons-paradox. A visual, interactive explanation of the phenomenon with a simple numerical example, useful for reinforcing intuition before applying it to real experimentation data. In English.
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — experimentguide.com. The chapter on segmentation covers how to detect and diagnose reversals like this lesson's in real product experiments, and why they almost always point to a problem in the experiment's design. In English.