Randomization

Overview

Lesson 3 deliberately left a question open: control and variant are already well defined, they run at the same time, they differ in exactly one change — but who decides which user goes into which group? This lesson answers that question with the piece that makes the whole design work: randomization — assigning each user to control or variant at random, with neither the user nor the team choosing, and with the assignment not depending on any user characteristic that could influence the result.

Randomization is the direct engineering answer to lesson 2's problem: if you assigned the most engaged users to variant and the least engaged to control —even by accident— any difference observed afterward would be contaminated by that prior difference between the groups, exactly like session time contaminated the correlation between seeing recommendations and buying. Random assignment is what guarantees that, on average, control and variant start out similar to each other in everything else —age, device, purchase intent, time of day they enter—, so the only systematic difference left between the two groups, by the end of the experiment, is the change being tested.

How this connects to the module. This lesson builds randomize(), the model you'll reuse in lesson 8's mini-project to design the real Mercado experiment's assignment. It also resolves a design decision that looks like a technical detail but is, in fact, as important as randomization itself: the randomization unit — what exactly is the thing being randomly assigned? The correct answer, almost always, is the user — never the individual session or request — and this lesson explains why.

An analogy: shuffling the cards well

Picture a 52-card deck representing every new Mercado user this week. You want to deal half the deck to control and the other half to variant, so both hands end up equally "strong" on average —neither with more high cards, nor with more of one particular suit—. If you shuffle the full deck well before dealing, and then give the first 26 cards to one player and the last 26 to the other, each hand ends up with a representative mix of high and low cards, of all four suits, with no systematic pattern favoring one hand over the other. That's, literally, what a good shuffle does: it eliminates any prior order or pattern in the deck, so each card's final position is unpredictable and independent of its characteristics.

Now imagine instead dealing the cards without shuffling — simply taking the deck as it came ordered from the factory (all cards of one suit together, in order) and giving the first half to one player. That player could end up with almost all the clubs and spades; the other, with almost all the hearts and diamonds. The two hands are no longer comparable — the difference between them doesn't come from the luck of the deal, it comes from there never having been a real shuffle. An A/B test with no real randomization —where, say, the first users to sign up go to control and the following ones to variant, with no shuffling at all— runs exactly that risk: any hidden pattern in the original order (users who arrived early could differ from those who arrived later, for whatever reason) gets trapped inside a single group, instead of spreading evenly between both.

Worked example: randomize() over 20 new Mercado users

Let's build a simple random-assignment model and verify, with real data, that it actually splits users in a balanced way. One important clarification first: this environment doesn't have Math.random() available, and even if it did, using a truly random number would make the experiment not reproducible —every time you ran the code, the same user could land in a different group—. Instead, we use a deterministic hash: a function that, from each user's id, always computes the same result. The same user always lands in the same group — and yet, the resulting assignment pattern behaves as if it were random, because it has no relationship to any real user characteristic.

// hashId: a simple, deterministic hash (NOT Math.random, which isn't available in
// this environment and would also break the experiment's reproducibility). It adds
// up the id's character codes and takes the remainder when divided by 2. The same
// id ALWAYS lands in the same group -- that's what makes the experiment reproducible
// and keeps the same user from seeing, across different visits, both control and variant.
function hashId(id) {
  let sum = 0;
  for (let i = 0; i < id.length; i++) sum += id.charCodeAt(i);
  return sum % 2;
}

// randomize: splits each user into control or variant based on their id's hash.
// The randomization unit is the USER (not the session or the request): each
// id appears only once in the input list, and therefore lands in a single
// group for the whole experiment.
function randomize(users) {
  const control = [];
  const variant = [];
  users.forEach((user) => {
    const bucket = hashId(user.id) === 0 ? control : variant;
    bucket.push(user);
  });
  return { control, variant };
}

// Pedagogical data: 20 new Mercado users who entered the week of the
// recommendations launch, with their type (new = first week at Mercado,
// returning = had bought before). The list's order has no relationship
// to each id's hash.
const users = [
  { id: 'mkt-u001', type: 'returning' },
  { id: 'mkt-u002', type: 'new' },
  { id: 'mkt-u003', type: 'returning' },
  { id: 'mkt-u004', type: 'returning' },
  { id: 'mkt-u005', type: 'new' },
  { id: 'mkt-u006', type: 'returning' },
  { id: 'mkt-u007', type: 'new' },
  { id: 'mkt-u008', type: 'returning' },
  { id: 'mkt-u009', type: 'returning' },
  { id: 'mkt-u010', type: 'new' },
  { id: 'mkt-u011', type: 'returning' },
  { id: 'mkt-u012', type: 'new' },
  { id: 'mkt-u013', type: 'returning' },
  { id: 'mkt-u014', type: 'returning' },
  { id: 'mkt-u015', type: 'new' },
  { id: 'mkt-u016', type: 'returning' },
  { id: 'mkt-u017', type: 'new' },
  { id: 'mkt-u018', type: 'returning' },
  { id: 'mkt-u019', type: 'new' },
  { id: 'mkt-u020', type: 'returning' },
];

console.log('=== randomize() over 20 new Mercado users ===\n');
const { control, variant } = randomize(users);

console.log('control (' + control.length + ' users): ' + control.map((u) => u.id).join(', '));
console.log('variant (' + variant.length + ' users): ' + variant.map((u) => u.id).join(', '));

function shareNew(group) {
  const newCount = group.filter((u) => u.type === 'new').length;
  return { newCount, total: group.length, pct: (newCount / group.length) * 100 };
}

const controlShare = shareNew(control);
const variantShare = shareNew(variant);

console.log('\nBalance of the "new vs returning" attribute (not used for randomization):');
console.log('  control: ' + controlShare.newCount + '/' + controlShare.total + ' new (' + controlShare.pct.toFixed(1) + '%)');
console.log('  variant: ' + variantShare.newCount + '/' + variantShare.total + ' new (' + variantShare.pct.toFixed(1) + '%)');

What to expect. When you run the file with Node, the output is exactly this:

=== randomize() over 20 new Mercado users ===

control (10 users): mkt-u002, mkt-u004, mkt-u006, mkt-u008, mkt-u011, mkt-u013, mkt-u015, mkt-u017, mkt-u019, mkt-u020
variant (10 users): mkt-u001, mkt-u003, mkt-u005, mkt-u007, mkt-u009, mkt-u010, mkt-u012, mkt-u014, mkt-u016, mkt-u018

Balance of the "new vs returning" attribute (not used for randomization):
  control: 4/10 new (40.0%)
  variant: 4/10 new (40.0%)

Notice the key result: hashId() never looked at any user's type field — it only used the id. And yet, the two groups ended up with exactly the same proportion of new users (40.0% in both). This isn't a coincidence of this particular set of 20 data points: it's the direct consequence of the hash having no relationship to type, so the mix of "new" and "returning" users spreads out, on average, just as evenly as the rest of the users' characteristics —age, device, purchase intent, and everything else not even in this table—. That's exactly what the well-shuffled-deck analogy predicts: no hand ends up systematically stronger than the other in any attribute, visible or invisible.

The randomization unit: why user, and not session or request

There's a design decision hiding behind the phrase "assign at random", one just as important as the shuffling itself: what exactly is the thing being assigned? That's called the randomization unit. For almost any product experiment, the correct answer is the user —identified stably, for example with their account id— never the session nor the individual request.

Why does it matter so much? Imagine that, instead of assigning by user, Mercado decided to randomize on every individual request —every time someone loads a product page, a new coin flip decides whether they see the carousel or not—. The same user, browsing normally during a single shopping session, could see the carousel on one product page and not see it on the next. That's called contamination: the same user ends up exposed to both versions of the experiment, so you can no longer meaningfully say "this user is control or variant" — in practice, they're both at once, and any measurement of whether they bought or not gets mixed between both exposures.

Randomizing by USER (correct):
  User u007 -> hashed once -> ALWAYS variant, in every future session
  ██████████████████████████████████████  (same experience, the whole experiment)

Randomizing by REQUEST (contaminated):
  User u007, request 1 -> control
  User u007, request 2 -> variant   <- the SAME user, the OTHER version
  User u007, request 3 -> control
  Is u007 control or variant? -- That question can no longer be answered.

With the randomization unit set to the user, hashId() calculates the bucket only once per id, and that same calculation repeats —deterministic, not truly random— every time that same user interacts with Mercado again. That's why today's deterministic hash isn't just a workaround for this environment's lack of Math.random(): it is, in fact, the correct pattern any real experimentation system uses in production, to guarantee the same user always lands in the same group.

Common mistakes

Randomizing by request or by session, contaminating the measurement. What happens: the assignment system flips a new coin every time the product page loads, instead of once per user, so the same user can see control on one visit and variant on the next. Why it happens: technically, it's simpler to decide "this time yes, this time no" on every page load than to build the infrastructure to stably remember which group each user belongs to over time. How to spot it: if you ask the system "which group does user u007 belong to?" and the answer changes depending on when you ask, the randomization unit is misdefined. How to fix it: as in today's randomize(), calculate the hash only once per user id —never per session or per request— and persist that assignment so it stays stable on every future visit from that same user.

Randomizing with a criterion that actually correlates with the outcome. What happens: someone proposes "let's assign to variant the users who enter at night, and to control those who enter during the day" —a rule that consistently splits users into two groups, but not at random—. Why it happens: any rule that divides users into two similarly sized halves superficially feels like valid randomization. How to spot it: ask yourself whether the assignment criterion (time of day, in this case) could, on its own, be related to purchasing behavior —nighttime buyers could have different spending habits than daytime ones—. If the answer is yes, that criterion isn't a good substitute for chance. How to fix it: always use a criterion, like today's id hash, that has no known or suspected relationship with the behavior you're measuring. A good identifier hash meets that condition; time of day, day of the week, or geographic region, almost never do.

Assuming "splitting 50/50" is already the same as "randomizing". What happens: a team assigns the first 6,000 users who sign up that week to control, and the next 6,000 to variant, satisfied because the result is a perfect 50/50 split. Why it happens: an exact 50/50 proportion looks, at a glance, just as "balanced" as a split produced by a real hash, and it's easy to confuse "equal sizes" with "random assignment". How to spot it: the split criterion is tied to arrival order —the first ones versus the last ones— not to a function independent of each user's characteristics. How to fix it: remember the unshuffled-deck analogy: splitting an ordered deck in half produces two equally sized hands, but very different in composition. The first users to sign up in a launch week (probably the ones paying closest attention to app updates) can systematically differ from the last ones. A 50/50 size is a necessary condition for a good experiment, but it isn't, on its own, evidence assignment was random.

Exercises

Exercise 1 — Calculate the hash by hand. Without running Node, calculate hashId('mkt-u099') following the same hashId() algorithm (add up the character codes and take the remainder when divided by 2). Useful character codes: m=109, k=107, t=116, -=45, u=117, 0=48, 9=57. Which group (control or variant) would that user land in?

See solution

Adding up mkt-u099's codes: m(109) + k(107) + t(116) + -(45) + u(117) + 0(48) + 9(57) + 9(57) = 656. 656 % 2 = 0. Since hashId(id) === 0 lands in control (per today's randomize() function: hashId(user.id) === 0 ? control : variant), user mkt-u099 would land in control.

Exercise 2 — Explain contamination with a concrete Mercado case. Imagine Mercado, by mistake, randomizes by session instead of by user: every time a user opens the app, there's a 50% chance of seeing the carousel, regardless of whether they saw it last time. A particular user opens the app five times in a week, sees the carousel in three of those five times, and ends up buying on the fifth visit (where they didn't see the carousel). How would you classify this user's purchase: as a control or variant conversion? Explain why this question, as posed, has no correct answer.

See solution

The question has no correct answer because, with session-based randomization, this user doesn't consistently belong to either group — they were exposed to both control (twice) and variant (three times) during the same measurement week. Attributing their final purchase to control because they didn't see the carousel that time ignores that they did see it on three previous visits, and those prior exposures could have influenced their final purchase decision. Attributing it to variant because "overall they saw the carousel more times than not" is an arbitrary rule no standard experimentation framework uses. The only real solution is the one today's lesson proposes: randomize by user, once, so this user would have seen (or not seen) the carousel consistently across their five visits — and their final purchase could be cleanly attributed to a single group.

Exercise 3 — Critique a proposed assignment scheme. A teammate proposes: "let's assign to variant the users whose username starts with letters A-M, and to control those starting with N-Z — it's easy to implement and gives an even split". What problem does this scheme have, compared to the id hash you used today?

See solution

The central problem is that a username's initial could correlate with the user's geographic or cultural origin (for example, certain names are more common in some regions than others), which would introduce a systematic difference between control and variant unrelated to chance but to that hidden variable — lesson 2's same third-variable problem, now hiding inside the assignment mechanism itself. Even though the effect is probably small in practice, the underlying point is that a name's initial isn't independent of user characteristics in the same guaranteed way a well-designed hash over an arbitrary identifier like mkt-u007 is — the hash has no conceptual relationship to anything about the user other than serving as a source of deterministic randomness. The general rule: any assignment criterion that could, even remotely, be related to the behavior being measured is a bad candidate for randomization.

Summary and next step

In this lesson you built and ran randomize(), which splits users into control or variant using a deterministic hash of their id —with no Math.random(), and yet with the same effect as a good card shuffle—. You verified, with Mercado's 20 users, that the result comes out balanced even in an attribute (new vs returning) the function never directly looked at. And you learned the design decision that makes that assignment mean something sustained over time: the randomization unit must be the user, never the session or the request, because any other choice risks contaminating the measurement —the same user exposed to both versions, with no way to attribute their behavior to either group.

Before moving on you should be able to: explain in your own words why a good id hash plays the same role as shuffling a deck of cards, identify when an assignment scheme is contaminated by an incorrect randomization unit, and spot an assignment criterion that "looks" random but actually correlates with something relevant.

With control, variant, and a valid randomization already in place, one last conceptual piece is missing before calculating any number: what exactly is the experiment assuming, before it runs? Lesson 5 introduces the null hypothesis — every A/B test's skeptical starting point.

Resources

• Statsig, "What are experimental units?" — statsig.com/perspectives/experimental-units-explained. Discusses the choice of randomization unit —user, device, session— and why that choice determines which questions the experiment can or can't answer. In English.
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing — experimentguide.com. The book's chapter on randomization develops, with more statistical depth, why random assignment is the piece that lets you talk about causality, and covers the contamination problem between units in detail. In English.
• Evan Miller, "How Not To Run An A/B Test" — evanmiller.org/how-not-to-run-an-ab-test.html. Although this article's main focus is peeking (a module 7 topic), its first section clearly explains why the integrity of the assignment process is the foundation any later analysis rests on. In English.