Relaunching, and verifying the lift holds

Description

Everything before this converges here. The v2 engine is validated in shadow mode (90.0% overall agreement and 84.6% at the cold-start segment, above modelMigrationDecision()'s two thresholds). The timeout with a fallback from the postmortem is already implemented. The ramp is still the same one this module's lesson 2 designed. This lesson relaunches recommendations from the canary, climbs the full ramp with rolloutPlan() — this time with latency fixed — and does something no previous lesson in this guide could do yet: verify, several weeks later, whether the original +18.75% lift holds, or whether it was just the novelty effect.

Connection to the module. This lesson reuses rolloutPlan() exactly as it stood in module 3, running it on the same four-stage ramp with already-corrected latency data. It also reuses noveltyCheck() exactly as it stood in module 6's lesson 7, without changing a line: it compares the lift measured in the first week after the relaunch against several weeks later, to tell a real, durable effect apart from a temporary spike of curiosity that deflates over time.

An analogy: the second takeoff, with the plane already inspected

Go back, one last time, to the flight analogy that opened this module. The first takeoff attempt — the original rollout — stopped mid-runway when the instruments confirmed a real fault: the latency turbulence. The ground crew investigated, found the exact cause, and fixed it — not with a vague promise of "it'll be better," but with a specific piece replaced (v1 for v2) and tested on the ground before trying again (the previous lesson's shadow mode).

This is the second takeoff. And this time, there's something more to verify besides whether the plane takes off without trouble: is the destination it reaches actually worth it, or did it just seem promising the first time someone saw it? A passenger who gets excited about a new destination the first week, and loses interest by the fourth, isn't the same signal as a passenger who keeps choosing that destination month after month. This lesson's noveltyCheck() is that second verification — not just "did we land well?", but "is it worth still going there?"

Worked example: clean relaunch and durability verification

// M8 L07: relaunches recommendations with v2 (latency fix validated in the
// previous lesson) on module 3's same ramp, and verifies with noveltyCheck()
// whether the +18.75% lift holds weeks later, or whether it was just a
// novelty effect. rolloutPlan() is EXACTLY module 3's. noveltyCheck() is
// EXACTLY module 6, lesson 7's -- same threshold (50%), same decayPct as a
// percentage, same input and return shape, with no change.

function rolloutPlan(stages) {
  const results = [];
  let halted = false;
  for (const s of stages) {
    if (halted) { results.push({ ...s, decision: 'NOT_REACHED' }); continue; }
    const decision = s.advanceIf(s.measured) ? 'ADVANCE' : 'HOLD';
    results.push({ ...s, decision });
    if (decision === 'HOLD') halted = true;
  }
  return results;
}
function noveltyCheck(weeklyLift) {
  const first = weeklyLift[0];
  const last = weeklyLift[weeklyLift.length - 1];
  const decayPct = Math.round(((first - last) / first) * 1000) / 10;
  const verdict = decayPct > 50 ? 'NOVELTY (fades)' : 'HOLDS (sustained)';
  return { weeklyLift, first, last, decayPct, verdict };
}

console.log('=== Part 1: rolloutPlan -- relaunch with v2 + latency fix ===\n');
const ceiling = 800;
const relaunchStages = [
  { percent: 0.01, label: 'canary 1%', measured: { p95Latency: 705 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.10, label: 'rollout 10%', measured: { p95Latency: 740 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 0.50, label: 'rollout 50%', measured: { p95Latency: 765 }, advanceIf: (m) => m.p95Latency <= ceiling },
  { percent: 1.00, label: 'rollout 100%', measured: { p95Latency: 778 }, advanceIf: (m) => m.p95Latency <= ceiling },
];
const relaunchResult = rolloutPlan(relaunchStages);
relaunchResult.forEach((s) => console.log(s.label.padEnd(14) + 'p95=' + s.measured.p95Latency + 'ms -> ' + s.decision));

console.log('\n=== Part 2: noveltyCheck -- does the +18.75% lift hold 4 weeks later? ===\n');
const weeklyLift = [0.1875, 0.1810, 0.1755, 0.1740];
weeklyLift.forEach((lift, i) => console.log('week ' + (i + 1) + ': +' + (lift * 100).toFixed(2) + '%'));
const novelty = noveltyCheck(weeklyLift);
console.log('\ndecay = (' + (novelty.first * 100).toFixed(2) + '% - ' + (novelty.last * 100).toFixed(2) + '%) / ' + (novelty.first * 100).toFixed(2) + '% = ' + novelty.decayPct + '%');
console.log(novelty.verdict);

What to expect. Running the file with Node, the output is exactly this:

=== Part 1: rolloutPlan -- relaunch with v2 + latency fix ===

canary 1%     p95=705ms -> ADVANCE
rollout 10%   p95=740ms -> ADVANCE
rollout 50%   p95=765ms -> ADVANCE
rollout 100%  p95=778ms -> ADVANCE

=== Part 2: noveltyCheck -- does the +18.75% lift hold 4 weeks later? ===

week 1: +18.75%
week 2: +18.10%
week 3: +17.55%
week 4: +17.40%

decay = (18.75% - 17.40%) / 18.75% = 7.2%
HOLDS (sustained)

Part 1 is the most visible difference from the first launch attempt: all four stages advance cleanly, ADVANCE on every one, all the way to 100% of Mercado's base. The exact same criterion (p95Latency <= 800) that stopped the ramp at 910ms during the first attempt now holds across all four stages — 705ms, 740ms, 765ms, 778ms — with the v2 engine keeping latency below the ceiling even at the highest-volume stage. Notice latency climbs a bit at each stage (more concurrent traffic, expectedly), but stays with margin against the ceiling across all four — unlike the first attempt, where the margin disappeared entirely at the second stage.

Part 2 answers a question no previous lesson in this entire guide could answer yet: was the original +18.75% a real effect, or the passing excitement of seeing something new? The lift drops, week over week — 18.75% → 18.10% → 17.55% → 17.40% — which is normal: some initial decay is almost always expected when part of the lift comes from the novelty of seeing something new for the first time. What matters is the magnitude of that drop: a 7.2% decay from the original lift, well below the 50% threshold noveltyCheck() uses to tell "a novelty effect that deflates" apart from "a real effect that holds" — the same threshold, with no adjustment, module 6's lesson 7 set in advance. The verdict is HOLDS (sustained): recommendations's gain doesn't depend on it being new — it keeps working once it stopped being new.

Why this verification didn't exist in any previous module

It's worth noting why noveltyCheck() shows up right here, and not earlier. Every previous piece of this guide — blastRadius(), isEnabled(), rolloutPlan(), guardrailWatch(), rollbackDecision() — works with data from one specific moment: the system's state at the instant it's measured. noveltyCheck() is different: it needs data from several weeks, because the question it answers — does this hold? — makes no sense with a single measurement. A launch can pass every latency guardrail, every ramp criterion, and still turn out, weeks later, to be an effect that deflates — that wouldn't be a failure of any previous piece, it would simply be a different kind of risk none of them is designed to catch. That's why a launch's full cycle doesn't end at "it reached 100% without breaking anything" — it ends when someone comes back, weeks later, and confirms the result is still real.

Common mistakes

Declaring the launch a success as soon as it reaches 100%, without waiting to measure durability. What happens: the team celebrates Part 1's result — four ADVANCEs in a row — and considers the work done there, without scheduling any follow-up measurement weeks later. Why it happens: reaching 100% with no HALT feels like the finish line, and it's easy to forget the experiment's real value — the sustained lift — hasn't been confirmed yet at that timescale. How to spot it: if nobody has a lift review scheduled several weeks after the relaunch, this lesson's Part 2 question is going to go unanswered. How to fix it: as in this lesson, a clean rollout through 100% is a necessary condition, but not a sufficient one — the cycle closes with noveltyCheck(), not before.

Interpreting any week-over-week drop as evidence the effect wasn't real. What happens: someone sees the lift dropped from 18.75% to 17.40% between week 1 and week 4, and concludes — incorrectly — "the result is falling apart, this isn't going to last." Why it happens: any number that drops intuitively feels like a bad sign, without comparing that drop's magnitude against a reasonable threshold. How to spot it: if the durability conclusion is based on "the number dropped" without calculating the decay percentage or comparing it against any threshold, the read is incomplete. How to fix it: as noveltyCheck() calculates, what matters isn't whether the lift dropped — it almost always drops a bit — but how much: a 7.2% decay, well below the 50% that would signal real concern, is consistent with a durable effect, not one that's deflating.

Relaunching with the original ramp without re-verifying the flag and its criteria. What happens: someone assumes that, since the flag and the ramp were already designed in this module's lesson 2, there's no need to review them again before this relaunch — just re-activate recommendationsFlag.enabled = true and climb straight up. Why it happens: the design work is already done, and repeating it feels redundant. How to spot it: if the relaunch doesn't include a new canary pass with real data from the v2 engine — as this lesson's Part 1 does, with p95=705ms at the first stage — it's being assumed, unverified, that the system's behavior with the new model is identical to what was measured with the old one. How to fix it: as in this lesson, the relaunch reruns the full ramp from the canary, with real data measured on the already-corrected system — no stage gets skipped just because the ramp "was already proven once" with a different engine.

Exercises

Exercise 1 — Calculate the decay for a real novelty case. Suppose that, for another Mercado feature (a limited-time discount banner), the measured lift was: week 1, 22%; week 4, 9%. Calculate decayPct with noveltyCheck()'s formula, and determine the verdict.

See solution

decayPct = Math.round(((0.22 - 0.09) / 0.22) * 1000) / 10 = 59.1. Since 59.1 > 50, the verdict would be NOVELTY (fades) — the original lift was probably inflated by the initial curiosity of seeing a new banner, and that effect mostly deflated by week 4. This result would be consistent with intuition: a "limited-time" banner is, by design, a type of feature more prone to generating an initial attention spike that doesn't hold, unlike a functional improvement like recommendations, which stays useful even once it stops being new.

Exercise 2 — Design a fifth verification week. Mercado's team wants to add a week 5 to this follow-up, to confirm the lift stays stable beyond the fourth week. If week 5 measured a lift of 17.20%, how would noveltyCheck()'s result change if that data point were added to the weeklyLift array?

See solution

With weeklyLift extended to five elements, noveltyCheck() would still compare weeklyLift[0] (week 1, 18.75%) against weeklyLift[weeklyLift.length - 1] — which would now be week 5 (17.20%), not week 4. The new decayPct would be Math.round(((0.1875 - 0.1720) / 0.1875) * 1000) / 10 = 8.3, still well below 50, so the verdict would still be HOLDS (sustained). This exercise confirms that noveltyCheck(), as written, always compares the first and last measurements of the array it receives — adding more intermediate weeks doesn't change the logic, it only updates which measurement counts as "the last" available.

Exercise 3 — Close the cycle in writing. Write the message (100-150 words) you'd send Mercado's executive team, four weeks after the relaunch, confirming recommendations is in production at 100% and the result holds. Include: the rollout's status, the latency with the fix applied, and noveltyCheck()'s result.

See solution

One possible message: "recommendations has been in production at 100% of our buyer base for four weeks now, with no additional incidents. The migration to the v2 engine resolved the latency problem that stopped the first attempt: p95Latency stays between 705ms and 778ms across the ramp's four stages, always below our 800ms ceiling, with margin. On the business result: checkout conversion lift, which started at +18.75% the first week, holds at +17.40% four weeks later — a drop of just 7.2%, within what's expected as an initial adjustment and far from indicating the effect was just passing curiosity. The result is real and durable. This closes the launch's full cycle: we caught the problem in time, fixed it with evidence, and confirmed the value holds." The message separates the technical status (rollout, latency) from the business result (durable lift), with exact numbers for both.

Summary and next step

In this lesson you relaunched recommendations with the already-validated v2 engine: the full ramp advances cleanly through 100%, with latency always below the 800ms ceiling across all four stages. You confirmed, with noveltyCheck(), that the +18.75% lift holds four weeks later (+17.40%, a decay of just 7.2%) — the result is durable, not a novelty effect that deflates.

Before moving on you should be able to: explain why noveltyCheck() needs data from several weeks, unlike the rest of this guide's pieces; and calculate decayPct by hand given two lift values.

With all seven pieces from M1 through M7 run, in order, on the same launch — from the blast radius to the durability verification — lesson 8 brings everything together into a single document: the complete launch plan, the end-to-end run in a single Node script, and the closing of this guide and the entire Product Engineering ecosystem.

Resources

• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments — experimentguide.com. Includes a detailed discussion of the novelty effect and why measuring a result only in the first week can be misleading. In English.
• Google SRE Workbook, Chapter 16, "Canarying Releases" — sre.google/workbook/canarying-releases. The formal reference for gradual rollout, now applied a second time in this guide on the same case, with the problem already fixed. In English.