Control and variant

Overview

With the underlying problem already clear —a correlation alone doesn't prove cause (lesson 2)— this lesson builds the solution's exact vocabulary: the two groups that make up every A/B test. control is the group that keeps living the usual experience, with no change at all. variant is the group that receives exactly one specific change —and only that change— on top of the same base experience. The comparison between what happens to control and what happens to variant, during the same time period, is the only thing that can isolate that specific change's effect.

That bolded phrase —during the same time period— isn't a minor detail. It's, together with lesson 4's randomization, one of the two conditions without which an A/B test stops being an A/B test and goes back to being, disguised with new names, the same correlation problem from the previous lesson.

How this connects to the module. This lesson defines the vocabulary (control, variant) you'll use unchanged for the rest of the module and the guide: lesson 4 explains how it's decided who goes into which group (at random), lesson 5 formalizes what the experiment assumes about the relationship between the two groups (the null hypothesis), and lesson 6 calculates the difference between their rates (the lift). Without control and variant well defined, none of those pieces has anything to lean on.

An analogy: the clinical trial, now in detail

Going back to lesson 1's clinical trial: control is the placebo group —it receives a pill identical in appearance, packaging, and administration to the real drug, but with no active ingredient—. variant is the treatment group —it receives exactly the drug being tested—. Notice a detail that's easy to miss: the placebo pill isn't "nothing". It's a carefully designed intervention meant to be identical in everything except the single variable the experiment wants to isolate. If the placebo group took a pill of a different color, or took it at a different time than the treatment group, the experiment would no longer be isolating the drug's effect — it would be mixing the drug's effect with the effect of the pill's color or the schedule.

The same applies, with the same rigor, to a product A/B test: control isn't "whatever was happening before"; it's a specific, stable experience, running at the same time as variant, and identical to variant in everything except the single change being tested. For Mercado: control is the usual product page, with no recommendations carousel. variant is exactly that same product page, with the same checkout, the same catalog, the same load speed —everything the same— except for one thing: the recommendations carousel shows up below the product description.

Worked example: the recommendations experiment's specification

Before running any experiment, a serious team writes out its full specification —what exactly control is, what exactly variant is, and what stays identical between the two—. For Mercado's recommendations launch, that specification looks like this:

Element	control	variant
Product page	Same as always	Same as always
Checkout	Same as always	Same as always
Product catalog	Same as always	Same as always
recommendations carousel	Doesn't appear	Does appear, below the description
Measurement period	Weeks 1-6 since launch	Weeks 1-6 since launch (the same)
Where users come from	New users entering that same week	New users entering that same week

The row deserving the most careful attention is "Measurement period": both groups run during exactly the same six weeks, not one after the other. That simultaneity is what rules out any alternative explanation tied to time —a marketing campaign that started that week, a sales season, a weather change that affects purchases— because, if something like that happened, it would affect control and variant equally, since both exist at the same moment.

                    Week  1    2    3    4    5    6
control  (no reco)  ████████████████████████████    <- runs the WHOLE window
variant  (with reco) ████████████████████████████    <- at the SAME TIME as control

                     Both groups, in parallel, the same 6 weeks.
                     Any external event (marketing, season) affects
                     BOTH equally -- it can't explain a difference
                     BETWEEN them.

Compare that design against a flawed one, very tempting because it looks simpler to implement: measure checkoutConversionRate during the six weeks before recommendations launches, and call that control; then measure the six weeks after the launch, and call that variant.

                    Week -6   -5   -4   -3   -2   -1  | 1    2    3    4    5    6
"control" (before)  ████████████████████████████      |
"variant" (after)                                      | ████████████████████████████

                     TWO DIFFERENT time periods, one after the other.
                     Anything that changed between those two periods
                     (season, prices, competition, weather) gets mixed
                     with recommendations's effect, with no way to separate them.

This second design —comparing "before" against "after"— isn't an A/B test, even though it sometimes gets mistakenly called one. It's exactly lesson 2's same problem, with a new layer of disguise: any observed difference between the two periods could be due to recommendations, or it could be due to anything else that changed between those two time windows —a sales season ending, a change in the competition, even the weather—. The only way to rule out those alternative explanations is for control and variant to run at the same time, exposed to exactly the same external context, and differ only in the single thing the experiment wants to measure.

Common mistakes

Comparing two different time periods (pre/post) instead of a simultaneous A/B. What happens: a team measures checkoutConversionRate six weeks before recommendations launches, and another six weeks after, and presents that comparison as if it were an A/B test. Why it happens: a pre/post design is easier to implement —no random-assignment infrastructure needed, just looking at the dashboard before and after a date— and "before vs. after" intuitively feels similar to "control vs. variant". How to spot it: control and variant don't correspond to the same calendar window. How to fix it: as in today's specification, control and variant must run simultaneously, exposed to the same external context; only then can a difference between them be attributed to the specific change, and not to whatever else changed over time.

Letting the user choose whether to be in control or variant. What happens: instead of assigning users, the team puts recommendations behind an optional toggle in the account settings, and compares those who turned it on (variant, by their own decision) against those who didn't (control, also by their own decision). Why it happens: giving the option to turn a feature on or off feels more respectful of the user, and technically seems simpler than building an assignment system. How to spot it: the decision of which group a user belongs to was made by the user themselves, not an external random system. How to fix it: users who proactively turn on a new feature are almost always, to begin with, different from those who don't —more curious, more engaged, more likely to buy anyway—, so any observed difference could be due to that self-selection bias, not the feature. Lesson 4 develops in depth why assignment has to be random, and never a user's choice.

Defining variant vaguely, without specifying exactly what changes. What happens: the experiment's specification just says "variant has the new, improved product experience", without detailing which specific component changed —is it the recommendations carousel? was the "add to cart" button also redesigned at the same time? did something change in checkout?—. Why it happens: it's more natural for the team to describe the whole new experience in general terms ("the improved version") than to isolate, with surgical precision, the single element that distinguishes variant from control. How to spot it: if you ask the team "what, exactly, does a variant user see that a control user doesn't?" and the answer mentions more than one change, or is imprecise about which one, the specification is incomplete. How to fix it: as in today's table, every row of the specification —except the carousel's— should explicitly say "same as always" in both columns. A well-defined A/B test can be summed up in a single precise sentence: "variant is identical to control, except for [this one change]".

Exercises

Exercise 1 — Find the design flaw. A teammate proposes this design for measuring recommendations: "let's measure checkoutConversionRate on the iOS app (which already has the carousel, launched last month) against checkoutConversionRate on the Android app (which doesn't have it yet, launching next month)". What simultaneity problem does this design have, even though the two groups run "at the same time" on the calendar?

See solution

Even though the two measurements happen in the same calendar period, iOS and Android aren't the same type of user — each platform's population can systematically differ in age, purchasing power, purchasing behavior, or any other characteristic tied to which operating system they use. This is, essentially, the same problem as comparing two different time periods, moved over to comparing two different populations: any observed difference between iOS and Android could be due to recommendations, or it could simply be that Mercado's iOS and Android users are, to begin with, different groups of people. Temporal simultaneity alone isn't enough; both groups also need to come from the same population, randomly assigned within it — lesson 4's exact topic.

Exercise 2 — Complete the specification. Using today's "Worked example" table's format, write the missing row for this situation: besides the recommendations carousel, Mercado wants to test whether changing the "Buy now" button from blue to orange improves conversion. Should that test run as part of the same recommendations experiment, or as a separate experiment? Justify with this lesson's vocabulary.

See solution

It should run as a separate experiment, with its own control/variant pair (blue button vs. orange button). If the button color change were added inside the same variant that already has the recommendations carousel, variant would stop differing from control in a single thing —it would stop being "identical except for the carousel"—, and any observed difference in checkoutConversionRate could no longer be cleanly attributed to either the carousel or the button color: the two changes would be mixed together, with no way to know which one (or whether both, or neither) caused the effect. This is exactly why an A/B test's specification requires isolating a single change per experiment.

Exercise 3 — Diagnose the pre/post design. A Mercado report says: "in October, before recommendations, checkoutConversionRate was 3.0%; in November, with recommendations already active for everyone, it rose to 3.9%. Conclusion: recommendations raises conversion by 30% relative." Using this lesson's vocabulary, what type of comparison is this, and what alternative explanation —unrelated to recommendations— could explain the rise from October to November?

See solution

It's a pre/post comparison, not an A/B test: there's no control running at the same time as variant — all of Mercado went from not having recommendations (October) to having them (November), with no simultaneous comparison group. A reasonable alternative explanation: November usually includes high-season shopping events (for example, year-end promotions many marketplaces launch around that time), which on their own could raise checkoutConversionRate with recommendations having nothing to do with it. Without a simultaneous control exposed to that same November, it's impossible to separate recommendations's effect from the season's effect — exactly the problem this design doesn't solve, and that a real A/B test, with both groups running the same time window, does solve.

Summary and next step

In this lesson you precisely defined every A/B test's central vocabulary: control (the usual experience) and variant (the same experience, with exactly one change added), and the condition that makes the comparison between them mean something: that they run simultaneously, exposed to the same external context. You saw, with the recommendations experiment's full specification, how that requirement gets documented in practice, and why a "before vs. after" design —tempting as it is for being simpler— isn't an A/B test.

Before moving on you should be able to: define control and variant in your own words, explain why simultaneity is a necessary condition (not just a recommended one), and detect when a comparison presented as "A/B" is actually a disguised pre/post design.

There's still one critical piece this lesson deliberately left open: how, exactly, is it decided which user goes into control and which into variant? Lesson 4 answers that question with the tool that makes everything above work: randomization.

Resources

• Optimizely, "A/B Testing" (optimization glossary) — optimizely.com/optimization-glossary/ab-testing. A clear summary of the control/variant framework (sometimes called treatment in other sources) and of a well-designed experiment's components. In English.
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — experimentguide.com. The book's introductory chapter defines, with the same rigor as this lesson, the difference between a real controlled experiment and a pre/post design, with examples of cases where that mistake cost important decisions. In English.
• Ron Kohavi and Stefan Thomke, "The Surprising Power of Online Experiments" (Harvard Business Review, Sept-Oct 2017) — hbr.org/2017/09/the-surprising-power-of-online-experiments. Includes real examples of why comparing different time periods led to wrong conclusions in large-scale experiments. In English.