Baseline retention: do people who buy come back?

Overview

The previous lesson confirmed recommendations converts more people in a single visit — or, at least, that it targets the funnel's correct step for trying to. But converting once and sticking around are different questions, and this lesson exists precisely to keep them from getting confused. Before looking at the A/B test's result, you're going to read Mercado's cohort retention: first the business's baseline, with no experiment involved, and then the descriptive comparison between control and variant — the same comparison module 3 already did, now a module later, as this capstone's full method's second layer.

How this connects to the module. This lesson reuses retentionCurve() and cohortTable() exactly as they stood in module 3's lessons 3 and 6, with no changes, over the same baseline and launch data you already saw in that module's project. It isn't an empty repeat: it's the layer-by-layer confirmation that the business behaves as expected before adding lesson 4's North Star and guardrails, and before this module's lessons 6 and 7's A/B test says whether lesson 2's conversion lift also translates into people coming back.

An analogy: the checkup before surgery

No surgeon operates without first checking the patient's resting vital signs — blood pressure, pulse, temperature, all before touching the scalpel. Not because they expect something to go wrong, but because without that reference point there's no way to know, after surgery, whether a change in vital signs was the intervention's effect or simply how that patient always behaves. Mercado's baseline cohort table is that prior checkup: before looking at whether recommendations changed anything, you need to know how Mercado's retention behaves when nobody touched anything.

Worked example: baseline, and control against variant

Part 1 — Mercado's natural retention, with no experiment involved

// Reusing retentionCurve() and cohortTable() EXACTLY as they stood in module 3's
// lessons 3 and 6, with no changes.
function retentionCurve(cohort, plateauThreshold = 2) {
  const initialSize = cohort[0];
  const rates = cohort.map((active) => (active / initialSize) * 100);
  let plateauStart = rates.length - 1;
  for (let week = rates.length - 1; week > 0; week--) {
    const delta = Math.abs(rates[week] - rates[week - 1]);
    if (delta <= plateauThreshold) {
      plateauStart = week - 1;
    } else {
      break;
    }
  }
  return { rates, plateauStart, plateauValue: rates[rates.length - 1] };
}

function cohortTable(cohorts) {
  const maxWeeks = Math.max(...cohorts.map((c) => c.active.length));
  const header = ['Cohort'.padEnd(12)]
    .concat(Array.from({ length: maxWeeks }, (_, w) => ('W' + w).padStart(6)))
    .join(' | ');
  console.log(header);
  console.log('-'.repeat(header.length));
  cohorts.forEach((c) => {
    const rates = c.active.map((a) => ((a / c.active[0]) * 100).toFixed(0) + '%');
    const cells = Array.from({ length: maxWeeks }, (_, w) => (rates[w] ? rates[w] : '--').padStart(6));
    console.log(c.week.padEnd(12) + ' | ' + cells.join(' | '));
  });
}

// Part 1: module 3's same baseline table -- four cohorts before the
// recommendations launch.
const baselineCohorts = [
  { week: '2026-06-01', active: [1000, 390, 300, 265, 255, 252] },
  { week: '2026-06-08', active: [1150, 445, 345, 305, 293] },
  { week: '2026-06-15', active: [980, 375, 290, 258] },
  { week: '2026-06-22', active: [1220, 470, 365] },
];

console.log('=== Part 1: Mercado\'s cohort table (baseline) ===\n');
cohortTable(baselineCohorts);
const baselineCurve = retentionCurve(baselineCohorts[0].active);
console.log('\nBaseline plateau (most mature cohort): ~' + baselineCurve.plateauValue.toFixed(1) + '%');

What to expect. Part 1's output:

=== Part 1: Mercado's cohort table (baseline) ===

Cohort       |     W0 |     W1 |     W2 |     W3 |     W4 |     W5
------------------------------------------------------------------
2026-06-01   |   100% |    39% |    30% |    27% |    26% |    25%
2026-06-08   |   100% |    39% |    30% |    27% |    25% |     --
2026-06-15   |   100% |    38% |    30% |    26% |     -- |     --
2026-06-22   |   100% |    39% |    30% |     -- |     -- |     --

Baseline plateau (most mature cohort): ~25.2%

Part 2 — control against variant, the launch week

// Part 2: control vs variant, exactly module 3's project's same numbers.
const control = [1000, 380, 290, 255, 248, 246];
const variant = [1000, 430, 350, 320, 310, 308];

console.log('\n=== Part 2: control vs variant ===\n');
const controlCurve = retentionCurve(control);
const variantCurve = retentionCurve(variant);
console.log('control (no recommendations): plateau ~' + controlCurve.plateauValue.toFixed(1) + '%');
console.log('variant (with recommendations): plateau ~' + variantCurve.plateauValue.toFixed(1) + '%');

const gap = variantCurve.plateauValue - controlCurve.plateauValue;
console.log('\nDescriptive plateau gap: +' + gap.toFixed(1) + ' points.');
console.log('This is NOT yet a statistically significant result.');

What to expect. When you run the full file (both parts together) with Node:

=== Part 1: Mercado's cohort table (baseline) ===

Cohort       |     W0 |     W1 |     W2 |     W3 |     W4 |     W5
------------------------------------------------------------------
2026-06-01   |   100% |    39% |    30% |    27% |    26% |    25%
2026-06-08   |   100% |    39% |    30% |    27% |    25% |     --
2026-06-15   |   100% |    38% |    30% |    26% |     -- |     --
2026-06-22   |   100% |    39% |    30% |     -- |     -- |     --

Baseline plateau (most mature cohort): ~25.2%

=== Part 2: control vs variant ===

control (no recommendations): plateau ~24.6%
variant (with recommendations): plateau ~30.8%

Descriptive plateau gap: +6.2 points.
This is NOT yet a statistically significant result.

Reading the result: the second layer confirms the first

This result isn't new — it's exactly the one module 3's project produced. What matters now is where it sits within this capstone's full method: Part 1 confirms control (24.6%) behaves the way Mercado always does, with no product change (baseline ~25.2%) — evidence the experiment's randomization didn't, by accident, produce a "weird" control group. Part 2 shows variant (30.8%) departs from that pattern, in the same positive direction you already saw in lesson 2's funnel: it doesn't just convert more people in a single visit, that same people seems to come back more often.

Notice the word that repeats, on purpose, in the code's output: descriptive. A +6.2-point difference between two cohorts of 1,000 users each is exactly the kind of number you still can't tell apart from natural variation between two different groups of people, without this module's lessons 6 and 7's significance tools. This lesson leaves accumulated two descriptive signals in the same direction —the funnel (lesson 2) and retention (this lesson)— but neither one, yet, with the statistical backing that's about to arrive in lesson 6.

Going deeper: why retention is the hardest signal to fake

Of the three signals this capstone accumulates before reaching the formal A/B test —funnel conversion, retention, and lesson 4's North Star— retention is, in a sense, the most honest. A recommendations carousel could raise single-visit conversion with tricks that generate no real value (an artificially attractive offer, minor checkout friction removed, a "what's this new thing" curiosity effect), and those tricks would show up in today's funnel without necessarily meaning a real product improvement. But it's much harder to fake retention with a cheap trick: if people come back week after week, it's because they found real value the first time, not because a shiny button pushed them into buying once. That's why retention complements the funnel: the funnel measures whether something convinces short term; retention measures whether that conviction holds up.

Common mistakes

Looking only at Part 2 and skipping Part 1's baseline. What happens: someone goes straight to comparing control against variant, without having first confirmed whether control's 24.6% is a "normal" number for Mercado or is itself already strange. Why it happens: the control-variant comparison feels like the part relevant to the business; checking the baseline feels like a dispensable preliminary step, the same trap module 3 already warned about. How to spot it: if nobody can say whether 24.6% is consistent with Mercado's historical behavior, Part 1 was never seriously run. How to fix it: as in this lesson, always confirm the baseline before interpreting any difference between an experiment's groups — it's the same discipline as the medical checkup before surgery.

Adding the retention signal and the funnel signal as if they were independent and "doubled" the confidence. What happens: the team reasons "the funnel improved AND retention improved, so we have double the evidence it works" — treating the two descriptive signals as if each one, separately, were already enough proof. Why it happens: two numbers pointing in the same direction feel more convincing together than apart, and it's easy to forget neither one has yet gone through a significance test. How to spot it: if the conversation says "we have two signals, it's already confirmed" with no p-value mentioned, confidence is being built on descriptive coincidence, not statistics. How to fix it: the two signals (funnel and retention) are valid reasons to take the hypothesis seriously and move forward with the formal A/B test — they aren't, on their own, the confirmation this module's lessons 6 and 7 provide.

Comparing variant's plateau against the historical baseline, instead of against control. What happens: someone compares 30.8% (variant) directly against ~25.2% (baseline), calculating a +5.6-point difference, instead of comparing variant against control from the experiment's same week (+6.2 points). Why it happens: the baseline is the "usual" number, and comparing against it feels more natural than comparing against a control group that ran simultaneously. How to spot it: if the report cites "+5.6 points against the historical baseline" instead of "+6.2 points against this week's control", the comparison is mixing two different time periods. How to fix it: an experiment's valid comparison is always against the control that ran at the same time as the variant — the historical baseline only serves to confirm that control behaves normally, not as the effect's own comparison point.

Exercises

Exercise 1 — Verify the baseline's stability in another column. Using Part 1's table, calculate the maximum difference between the four cohorts at column W1. Is it consistent with the stability you already saw at W2 in module 3's project?

See solution

At W1, the four cohorts show: 39%, 39%, 38%, 39% — a maximum difference of just 1 percentage point. It's just as stable as column W2 (where all four gave exactly 30%), confirming the same pattern: Mercado's natural retention, with no product change, moves very little from one cohort to the next. This makes it more notable, by contrast, that variant departed 6.2 points from its corresponding control — a gap far bigger than the natural variation between normal cohorts.

Exercise 2 — Connect retention with lesson 2's funnel. If lesson 2's hypothesis is correct —recommendations reduces the leak at view_product → add_to_cart—, why might that improvement, happening in a single visit, also translate into a retention improvement weeks later? Propose a plausible mechanism.

See solution

A plausible mechanism: if the carousel helps people find, on their first visit, a product they're truly interested in (instead of leaving with nothing added to the cart), that successful first purchase increases the likelihood the person trusts Mercado as a place where they "find what they're looking for" — and that trust is precisely what brings people back the following week. The funnel measures the first effect (less leak, today); retention measures whether that first effect turns into a habit (people come back, later). It isn't a guaranteed mechanism —it might not hold up— but it's the reason it makes sense, from the start, to expect both signals to move together, as they in fact did in today's data.

Exercise 3 — Design the missing question. With the funnel (lesson 2) and retention (this lesson) already confirmed in the same positive direction, what specific question is still missing before deciding to ship recommendations to all users? It isn't a significance question yet — think about what else might be "paying" for that improvement.

See solution

The missing question is, exactly, this module's lesson 4's: what did this improvement in conversion and retention cost, in other metrics? An improvement in the metric being optimized should always be checked against the guardrails protecting the rest of the business —did latency get worse? did complaints rise? did margin drop?— exactly the same Goodhart's law argument you already saw in module 4. Two positive signals (funnel and retention) don't rule out a hidden cost elsewhere; only this capstone's lesson 4, with guardrailCheck(), can confirm or rule that out.

Summary and next step

In this lesson you confirmed, with module 3's same cohort retention, that the method's second layer points in the same direction as the first: Mercado's baseline plateau (~25.2%) is stable, this experiment's control behaves normally (24.6%), and the variant that saw recommendations stabilizes notably higher (30.8%), a descriptive difference of +6.2 points that, just like lesson 2's funnel result, still hasn't gone through any significance test.

Where you go next. Lesson 4 goes up a level: with two positive descriptive signals accumulated (funnel and retention), it's time to ask what achieving them might have cost. You're going to reuse module 4's evaluateNorthStar() and guardrailCheck() to confirm Mercado's official North Star and check whether any of its guardrails —latency, complaints, churn, margin— broke while recommendations ran.

Resources

• Amplitude, "What Is Cohort Retention Analysis: Essential Metrics Guide" — amplitude.com/explore/analytics/cohort-retention-analysis. Module 3's same base resource, worth rereading now that retention is read as a second layer within this capstone's full method. In English.
• Casey Winters and Lenny Rachitsky, "What Is Good Retention: An Exhaustive Benchmark Study" — lennysnewsletter.com/p/what-is-good-retention-issue-29. Still the reference for contrasting Mercado's plateau (~25% in control, ~31% in variant) against real industry benchmarks. In English.
• Brian Balfour (Reforge), "The Retention Engagement Growth Silent Killer" — reforge.com/blog/retention-engagement-growth-silent-killer. Argues why a conversion improvement with no accompanying retention improvement is usually a fragile win — the same argument motivating bringing both signals together in this capstone. In English.